Azure develop solutions Blob Storage

The following is from Azure Developer Training lab for AZ-203

Introduction to Azure Blob storage

Azure Blob storage is Microsoft’s object storage solution for the cloud. Blob storage is optimized for storing massive amounts of unstructured data. Unstructured data is data that does not adhere to a particular data model or definition, such as text or binary data.

About Blob storage

Blob storage is designed for:

  • Serving images or documents directly to a browser.
  • Storing files for distributed access.
  • Streaming video and audio.
  • Writing to log files.
  • Storing data for backup and restore, disaster recovery, and archiving.
  • Storing data for analysis by an on-premises or Azure-hosted service.

Users or client applications can access objects in Blob storage via HTTP/HTTPS, from anywhere in the world. Objects in Blob storage are accessible via the Azure Storage REST API, Azure PowerShell, Azure CLI, or an Azure Storage client library. Clientlibraries are available for a variety of languages, including .NET, Java, Node.js, Python, Go, PHP, and Ruby.

About Azure Data Lake Storage Gen2

Blob storage supports Azure Data Lake Storage Gen2, Microsoft’s enterprise big data analytics solution for the cloud. Azure Data Lake Storage Gen2 offers a hierarchical file system as well as the advantages of Blob storage, including low-cost, tiered storage;high availability; strong consistency; and disaster recovery capabilities.

Blob storage resources

Blob storage offers three types of resources:

  • The storage account.
  • A container in the storage account
  • A blob in a container

The following diagram shows the relationship between these resources.

Storage accounts

A storage account provides a unique namespace in Azure for your data. Every object that you store in Azure Storage has an address that includes your unique account name. The combination of the account name and the Azure Storage service endpoint forms theendpoints for your storage account.

For example, if your storage account is named mystorageaccount, then the default endpoint for Blob storage is:

http://mystorageaccount.blob.core.windows.net

To learn more about storage accounts, see Azure storage account overview.

Containers

A container organizes a set of blobs, similar to a directory in a file system. A storage account can include an unlimited number of containers, and a container can store an unlimited number of blobs.

Note: The container name must be lowercase.

Blobs

Azure Storage supports three types of blobs:

  • Block blobs store text and binary data, up to about 4.7 TB. Block blobs are made up of blocks of data that can be managed individually.
  • Append blobs are made up of blocks like block blobs, but are optimized for append operations. Append blobs are ideal for scenarios such as logging data from virtual machines.
  • Page blobs store random access files up to 8 TB in size. Page blobs store the virtual hard drive (VHD) files serve as disks for Azure virtual machines.

Move data to Blob storage

A number of solutions exist for migrating existing data to Blob storage:

  • AzCopy is an easy-to-use command-line tool for Windows and Linux that copies data to and from Blob storage, across containers, or across storage accounts.
  • The Azure Storage Data Movement library is a .NET library for moving data between Azure Storage services. The AzCopy utility is built with the Data Movement library.
  • Azure Data Factory supports copying data to and from Blob storage by using the account key, shared access signature, service principal, or managed identities for Azure resources authentications.
  • Blobfuse is a virtual file system driver for Azure Blob storage. You can use blobfuse to access your existing block blob data in your Storage account through the Linux file system.
  • Azure Data Box Disk is a service for transferring on-premises data to Blob storage when large datasets or network constraints make uploading data over the wire unrealistic. You can use Azure Data Box Disk to request solid-state disks (SSDs) fromMicrosoft. You can then copy your data to those disks and ship them back to Microsoft to be uploaded into Blob storage.
  • The Azure Import/Export service provides a way to export large amounts of data from your storage account to hard drives that you provide and that Microsoft then ships back to you with your data.

 

Azure Blob storage tiers

Overview

Azure storage offers different storage tiers which allow you to store Blob object data in the most cost-effective manner. The available tiers include:

  • Hot storage: is optimized for storing data that is accessed frequently.
  • Cool storage is optimized for storing data that is infrequently accessed and stored for at least 30 days.
  • Archive storage is optimized for storing data that is rarely accessed and stored for at least 180 days with flexible latency requirements (on the order of hours).

The following considerations accompany the different storage tiers:

  • The Archive storage tier is only available at the blob level and not at the storage account level.
  • Data in the Cool storage tier can tolerate slightly lower availability, but still requires high durability and similar time-to-access and throughput characteristics as Hot data. For Cool data, a slightly lower availability SLA and higher access costs compared toHot data are acceptable trade-offs for lower storage costs.
  • Archive storage is offline and offers the lowest storage costs but also the highest access costs.
  • Only the Hot and Cool storage tiers can be set at the account level. Currently the Archive tier cannot be set at the account level.
  • Hot, Cool, and Archive tiers can be set at the object level.

Data stored in the cloud grows at an exponential pace. To manage costs for your expanding storage needs, it’s helpful to organize your data based on attributes like frequency-of-access and planned retention period to optimize costs. Data stored in the cloud canbe different in terms of how it is generated, processed, and accessed over its lifetime. Some data is actively accessed and modified throughout its lifetime. Some data is accessed frequently early in its lifetime, with access dropping drastically as the data ages. Somedata remains idle in the cloud and is rarely, if ever, accessed once stored.

Each of these data access scenarios benefits from a different storage tier that is optimized for a particular access pattern. With Hot, Cool, and Archive storage tiers, Azure Blob storage addresses this need for differentiated storage tiers with separate pricingmodels.

Storage accounts that support tiering

You may only tier your object storage data to Hot, Cool, or Archive in Blob storage or General Purpose v2 (GPv2) accounts. General Purpose v1 (GPv1) accounts do not support tiering. However, customers can easily convert their existing GPv1 or Blob storageaccounts to GPv2 accounts through a simple one-click process in the Azure portal. GPv2 provides a new pricing structure for blobs, files, and queues, and access to a variety of other new storage features as well. Furthermore, going forward some new featuresand prices cuts will only be offered in GPv2 accounts. Therefore, customers should evaluate using GPv2 accounts but only use them after reviewing the pricing for all services as some workloads can be more expensive on GPv2 than GPv1.

Blob storage and GPv2 accounts expose the Access Tier attribute at the account level, which allows you to specify the default storage tier as Hot or Cool for any blob in the storage account that does not have an explicit tier set at the object level. For objects withthe tier set at the object level, the account tier will not apply. The Archive tier can only be applied at the object level. You can switch between these storage tiers at any time.

Hot access tier

Hot storage has higher storage costs than Cool and Archive storage, but the lowest access costs. Example usage scenarios for the Hot storage tier include:

  • Data that is in active use or expected to be accessed (read from and written to) frequently.
  • Data that is staged for processing and eventual migration to the Cool storage tier.

Cool access tier

Cool storage tier has lower storage costs and higher access costs compared to Hot storage. This tier is intended for data that will remain in the Cool tier for at least 30 days. Example usage scenarios for the Cool storage tier include:

  • Short-term backup and disaster recovery datasets.
  • Older media content not viewed frequently anymore but is expected to be available immediately when accessed.
  • Large data sets that need to be stored cost effectively while more data is being gathered for future processing. (For example, long-term storage of scientific data, raw telemetry data from a manufacturing facility)

Archive access tier

Archive storage has the lowest storage cost and higher data retrieval costs compared to Hot and Cool storage. This tier is intended for data that can tolerate several hours of retrieval latency and will remain in the Archive tier for at least 180 days.

While a blob is in Archive storage, it is offline and cannot be read (except the metadata, which is online and available), copied, overwritten, or modified. Nor can you take snapshots of a blob in Archive storage. However, you may use existing operations to delete,list, get blob properties/metadata, or change the tier of your blob.

Example usage scenarios for the Archive storage tier include:

  • Long-term backup, secondary backup, and archival datasets
  • Original (raw) data that must be preserved, even after it has been processed into final usable form. (For example, Raw media files after transcoding into other formats)
  • Compliance and archival data that needs to be stored for a long time and is hardly ever accessed. (For example, Security camera footage, old X-Rays/MRIs for healthcare organizations, audio recordings, and transcripts of customer calls for financialservices)
Blob rehydration

To read data in Archive storage, you must first change the tier of the blob to Hot or Cool. This process is known as rehydration and can take up to 15 hours to complete. Large blob sizes are recommended for optimal performance. Rehydrating several small blobsconcurrently may add additional time.

During rehydration, you may check the Archive Status blob property to confirm if the tier has changed. The status reads “rehydrate-pending-to-hot” or “rehydrate-pending-to-cool” depending on the destination tier. Upon completion, the Archive status propertyis removed, and the Access Tier blob property reflects the new Hot or Cool tier.

 

Comparison of the storage tiers

The following table shows a comparison of the Hot, Cool, and Archive storage tiers.

Hot storage tier Cool storage tier Archive storage tier
Availability 99.9% 99% N/A
Availability
(RA-GRS reads) 99.99% 99.9% N/A
Usage charges Higher storage costs, lower access and transaction costs Lower storage costs, higher access and transaction costs Lowest storage costs, highest access and transaction costs
Minimum object size N/A N/A N/A
Minimum storage duration N/A 30 days (GPv2 only) 180 days
Latency
(Time to first byte) milliseconds milliseconds < 15 hrs
Scalability and performance targets Same as general-purpose storage accounts Same as general-purpose storage accounts Same as general-purpose storage accounts

Understanding Block Blobs, Append Blobs, and Page Blobs

he storage service offers three types of blobs, block blobs, append blobs, and page blobs. You specify the blob type when you create the blob. Once the blob has been created, its type cannot be changed, and it can be updated only by using operations appropriatefor that blob type, i.e., writing a block or list of blocks to a block blob, appending blocks to a append blob, and writing pages to a page blob.

All blobs reflect committed changes immediately. Each version of the blob has a unique tag, called an ETag, that you can use with access conditions to assure you only change a specific instance of the blob.

Any blob can be leased for exclusive write access. When a blob is leased, only calls that include the current lease ID can modify the blob or (for block blobs) its blocks. Any blob can be duplicated in a snapshot.

Note: Blobs in the Azure storage emulator are limited to a maximum size of 2 GB.

About Block Blobs

Block blobs let you upload large blobs efficiently. Block blobs are comprised of blocks, each of which is identified by a block ID. You create or modify a block blob by writing a set of blocks and committing them by their block IDs. Each block can be a differentsize, up to a maximum of 100 MB (4 MB for requests using REST versions before 2016-05-31), and a block blob can include up to 50,000 blocks. The maximum size of a block blob is therefore slightly more than 4.75 TB (100 MB X 50,000 blocks). For RESTversions before 2016-05-31, the maximum size of a block blob is a little more than 195 GB (4 MB X 50,000 blocks). If you are writing a block blob that is no more than 256 MB (64 MB for requests using REST versions before 2016-05-31) in size, you can upload itin its entirety with a single write operation.

Storage clients default to a 128 MB maximum single blob upload, settable using the SingleBlobUploadThresholdInBytes property of the BlobRequestOptions object. When a block blob upload is larger than the value in this property, storageclients break the file into blocks. You can set the number of threads used to upload the blocks in parallel on a per-request basis using the ParallelOperationThreadCount property of the BlobRequestOptions object.

When you upload a block to a blob in your storage account, it is associated with the specified block blob, but it does not become part of the blob until you commit a list of blocks that includes the new block’s ID. New blocks remain in an uncommitted state untilthey are specifically committed or discarded. Writing a block does not update the last modified time of an existing blob.

Block blobs include features that help you manage large files over networks. With a block blob, you can upload multiple blocks in parallel to decrease upload time. Each block can include an MD5 hash to verify the transfer, so you can track upload progress andre-send blocks as needed. You can upload blocks in any order, and determine their sequence in the final block list commitment step. You can also upload a new block to replace an existing uncommitted block of the same block ID. You have one week to commitblocks to a blob before they are discarded. All uncommitted blocks are also discarded when a block list commitment operation occurs but does not include them.

You can modify an existing block blob by inserting, replacing, or deleting existing blocks. After uploading the block or blocks that have changed, you can commit a new version of the blob by committing the new blocks with the existing blocks you want to keepusing a single commit operation. To insert the same range of bytes in two different locations of the committed blob, you can commit the same block in two places within the same commit operation. For any commit operation, if any block is not found, the entirecommitment operation fails with an error, and the blob is not modified. Any block commitment overwrites the blob’s existing properties and metadata, and discards all uncommitted blocks.

Block IDs are strings of equal length within a blob. Block client code usually uses base-64 encoding to normalize strings into equal lengths. When using base-64 encoding, the pre-encoded string must be 64 bytes or less. Block ID values can be duplicated indifferent blobs. A blob can have up to 100,000 uncommitted blocks, but their total size cannot exceed 200,000 MB.

If you write a block for a blob that does not exist, a new block blob is created, with a length of zero bytes. This blob will appear in blob lists that include uncommitted blobs. If you don’t commit any block to this blob, it and its uncommitted blocks will be discardedone week after the last successful block upload. All uncommitted blocks are also discarded when a new blob of the same name is created using a single step (rather than the two-step block upload-then-commit process).

About Page Blobs

Page blobs are a collection of 512-byte pages optimized for random read and write operations. To create a page blob, you initialize the page blob and specify the maximum size the page blob will grow. To add or update the contents of a page blob, you write apage or pages by specifying an offset and a range that align to 512-byte page boundaries. A write to a page blob can overwrite just one page, some pages, or up to 4 MB of the page blob. Writes to page blobs happen in-place and are immediately committed to theblob. The maximum size for a page blob is 8 TB.

About Append Blobs

An append blob is comprised of blocks and is optimized for append operations. When you modify an append blob, blocks are added to the end of the blob only, via the Append Block operation. Updating or deleting of existing blocks is not supported. Unlike ablock blob, an append blob does not expose its block IDs.

Each block in an append blob can be a different size, up to a maximum of 4 MB, and an append blob can include up to 50,000 blocks. The maximum size of an append blob is therefore slightly more than 195 GB (4 MB X 50,000 blocks).

Reacting to Blob storage events

Azure Storage events allow applications to react to the creation and deletion of blobs using modern serverless architectures. It does so without the need for complicated code or expensive and inefficient polling services. Instead, events are pushed through AzureEvent Grid to subscribers such as Azure Functions, Azure Logic Apps, or even to your own custom http listener, and you only pay for what you use.

Blob storage events are reliably sent to the Event grid service which provides reliable delivery services to your applications through rich retry policies and dead-letter delivery.

Common Blob storage event scenarios include image or video processing, search indexing, or any file-oriented workflow. Asynchronous file uploads are a great fit for events. When changes are infrequent, but your scenario requires immediate responsiveness,event-based architecture can be especially efficient.

 

Blob storage accounts

Blob storage events are available in general-purpose v2 storage accounts and Blob storage accounts. General-purpose v2 storage accounts support all features for all storage services, including Blobs, Files, Queues, and Tables. A Blob storage account is a specialized storage account for storing your unstructured data as blobs (objects) in Azure Storage. Blob storage accounts are like general-purpose storage accounts and share all the great durability, availability, scalability, and performance features that you use today including 100% API consistency for block blobs and append blobs.

Available Blob storage events

Event grid uses event subscriptions to route event messages to subscribers. Blob storage event subscriptions can include two types of events:

Event Name Description
Microsoft.Storage.BlobCreated Fired when a blob is created or replaced through the PutBlob, PutBlockList, or CopyBlob operations
Microsoft.Storage.BlobDeleted Fired when a blob is deleted through a DeleteBlob operation

Event Schema

Blob storage events contain all the information you need to respond to changes in your data. You can identify a Blob storage event because the eventType property starts with “Microsoft.Storage”. Additional information about the usage of Event Grid event properties is documented in Event Grid event schema.

Filtering events

Blob event subscriptions can be filtered based on the event type and by the container name and blob name of the object that was created or deleted. Filters can be applied to event subscriptions either during the creation of the event subscription or at a later time.Subject filters in Event Grid work based on “begins with” and “ends with” matches, so that events with a matching subject are delivered to the subscriber. The subject of Blob storage events uses the format:

/blobServices/default/containers/<containername>/blobs/<blobname>

To match all events for a storage account, you can leave the subject filters empty. To match events from blobs created in a set of containers sharing a prefix, use a subjectBeginsWith filter like:

/blobServices/default/containers/containerprefix

To match events from blobs created in specific container, use a subjectBeginsWith filter like:

/blobServices/default/containers/containername/

To match events from blobs created in specific container sharing a blob name prefix, use a subjectBeginsWith filter like:

/blobServices/default/containers/containername/blobs/blobprefix

To match events from blobs created in specific container sharing a blob suffix, use a subjectEndsWith filter like “.log” or “.jpg”.

Practices for consuming events

Applications that handle Blob storage events should follow a few recommended practices:

  • As multiple subscriptions can be configured to route events to the same event handler, it is important not to assume events are from a particular source, but to check the topic of the message to ensure that it comes from the storage account you are expecting.
  • Similarly, check that the eventType is one you are prepared to process, and do not assume that all events you receive will be the types you expect.
  • As messages can arrive out of order and after some delay, use the etag fields to understand if your information about objects is still up-to-date. Also, use the sequencer fields to understand the order of events on any particular object.
  • Use the blobType field to understand what type of operations are allowed on the blob, and which client library types you should use to access the blob. Valid values are either BlockBlob or PageBlob.
  • Use the url field with the CloudBlockBlob and CloudAppendBlob constructors to access the blob.
  • Ignore fields you don’t understand. This practice will help keep you resilient to new features that might be added in the future.

 

Shared Access Signatures

A Shared Access Signature (SAS) is a URI that grants restricted access rights to containers, binary large objects (blobs), queues, and tables for a specific time interval. By providing a client with a Shared Access Signature, you can enable them to access resources in your storage account without sharing your account key with them.

The Shared Access Signature URI query parameters incorporate all of the information necessary to grant controlled access to a storage resource. The URI query parameters specify the time interval over which the Shared Access Signature is valid, the permissions that it grants, the resource that is to be made available, and the signature that the storage services should use to authenticate the request.

Here is an example of a SAS URI that provides read and write permissions to a blob. The table breaks down each part of the URI to understand how it contributes to the SAS:

https://myaccount.blob.core.windows.net/sascontainer/sasblob.txt?sv=2012-02-12&st=2013-04-29T22%3A18%3A26Z&se=2013-04-30T02%3A23%3A26Z&sr=b&sp=rw&sig=Z%2FRHIX5Xcg0Mq2rqI3OlWTjEg2tYkboXr1P9ZUXDtkk%3

Component Content Description
Blob URI https://myaccount.blob.core.windows.net/sascontainer/sasblob.txt The address of the blob. Note that using HTTPS is highly recommended.
Storage services version sv=2012-02-12 For Azure Storage services version 2012-02-12 and later, this parameter indicates the version to use.
Start time st=2013-04-29T22%3A18%3A26Z Specified in an International Organization for Standardization (ISO) 8061 format. If you want the SAS to be valid immediately, omit the start time.
Expiration time se=2013-04-30T02%3A23%3A26Z Specified in an ISO 8061 format.
Resource sr=b The resource is a blob.
Permissions sp=rw The permissions granted by the SAS include Read (r) and Write (w).
Signature sig=Z%2FRHIX5Xcg0Mq2rqI3OlWTjEg2tYkboXr1P9ZUXDtkk%3D Used to authenticate access to the blob. The signature is a HMAC function computed over a string to sign and a key by using the SHA256 algorithm and then encoded by using Base64 encoding.

Valet key pattern that uses Shared Access Signatures

A common scenario where an SAS is useful is a service where users read and write their own data to your storage account. In a scenario where a storage account stores user data, there are two typical design patterns:

  • Clients upload and download data via a front-end proxy service, which performs authentication. This front-end proxy service has the advantage of allowing the validation of business rules, but for large amounts of data or high-volume transactions, creating aservice that can scale to match demand might be expensive or difficult.
  • Using the valet key pattern, a lightweight service authenticates the client as needed and then generates an SAS. After the client receives the SAS, they can access storage account resources directly with the permissions defined by the SAS and for the intervalallowed by the SAS. The SAS mitigates the need for routing all data through the front-end proxy service.

 

Stored access policies

A Shared Access Signature can take one of two forms:

  • An ad hoc SAS. When you create an ad hoc SAS, the start time, expiration time, and permissions for the SAS are all specified on the SAS URI (or implied in the case where the start time is omitted). This type of SAS can be created on a container, blob, table,or queue.
  • An SAS with a stored access policy. A stored access policy is defined on a resource container—a blob container, table, or queue—and can be used to manage constraints for one or more Shared Access Signatures. When you associate an SAS with a storedaccess policy, the SAS inherits the constraints—the start time, expiration time, and permissions—defined for the stored access policy.

The difference between the two forms is important for one key scenario: revocation. An SAS is a URL, so anyone who obtains the SAS can use it regardless of who requested it to begin with. If an SAS is published publicly, it can be used by anyone in the world.Stored access policies give you the option to revoke permissions without having to regenerate the storage account keys. Set the expiration on these to be a very long time (or infinite), and make sure that it is regularly updated to move it further into the future.

 

Working with Blob Storage

Setting and Retrieving Properties and Metadata for Blob Resources by using REST

Containers and blobs support custom metadata, represented as HTTP headers. Metadata headers can be set on a request that creates a new container or blob resource, or on a request that explicitly creates a property on an existing resource.

Metadata Header Format

Metadata headers are name/value pairs. The format for the header is:

x-ms-meta-name:string-value

Beginning with version 2009-09-19, metadata names must adhere to the naming rules for C# identifiers.

Names are case-insensitive. Note that metadata names preserve the case with which they were created, but are case-insensitive when set or read. If two or more metadata headers with the same name are submitted for a resource, the Blob service returns statuscode 400 (Bad Request).

The metadata consists of name/value pairs. The total size of all metadata pairs can be up to 8KB in size.

Metadata name/value pairs are valid HTTP headers, and so they adhere to all restrictions governing HTTP headers.

Operations on Metadata

Metadata on a blob or container resource can be retrieved or set directly, without returning or altering the content of the resource.

Note that metadata values can only be read or written in full; partial updates are not supported. Setting metadata on a resource overwrites any existing metadata values for that resource.

Retrieving Properties and Metadata

The GET and HEAD operations both retrieve metadata headers for the specified container or blob. These operations return headers only; they do not return a response body. The URI syntax for retrieving metadata headers on a container is as follows:

GET/HEAD https://myaccount.blob.core.windows.net/mycontainer?restype=container

The URI syntax for retrieving metadata headers on a blob is as follows:

GET/HEAD https://myaccount.blob.core.windows.net/mycontainer/myblob?comp=metadata

Setting Metadata Headers

The PUT operation sets metadata headers on the specified container or blob, overwriting any existing metadata on the resource. Calling PUT without any headers on the request clears all existing metadata on the resource.

The URI syntax for setting metadata headers on a container is as follows:

PUT https://myaccount.blob.core.windows.net/mycontainer?comp=metadata?restype=container

The URI syntax for setting metadata headers on a blob is as follows:

PUT https://myaccount.blob.core.windows.net/mycontainer/myblob?comp=metadata

Standard HTTP Properties for Containers and Blobs

Containers and blobs also support certain standard HTTP properties. Properties and metadata are both represented as standard HTTP headers; the difference between them is in the naming of the headers. Metadata headers are named with the header prefix x-ms-meta- and a custom name. Property headers use standard HTTP header names, as specified in the Header Field Definitions section 14 of the HTTP/1.1 protocol specification.

The standard HTTP headers supported on containers include:

  • ETag
  • Last-Modified

The standard HTTP headers supported on blobs include:

  • ETag
  • Last-Modified
  • Content-Length
  • Content-Type
  • Content-MD5
  • Content-Encoding
  • Content-Language
  • Cache-Control
  • Origin
  • Range

 

Manipulating blob container properties in .NET

The CloudStorageAccount class contains the CreateCloudBlobClient method that gives you programmatic access to a client that manages your file shares:

CloudBlobClient client = storageAccount.CreateCloudBlobClient();

To reference a specific blob container, you can use the GetContainerReference method of the CloudBlobClient class:

CloudBlobContainer container = client.GetContainerReference(“images”);

After you have a reference to the container, you can ensure that the container exists. This will create the container if it does not already exist in the Azure storage account:

container.CreateIfNotExists();

With a hydrated reference, you can perform actions such as fetching the properties (metadata) of the container by using the FetchAttributesAsync method of the CloudBlobContainer class:

await container.FetchAttributesAsync();

After the method is invoked, the local variable is hydrated with values for various container metadata. This metadata can be accessed by using the Properties property of the CloudBlobContainer class, which is of typeBlobContainerProperties:

container.Properties

This class has properties that can be set to change the container, including (but not limited to) those in the following table.

Property Description
ETag This is a standard HTTP header that gives a value that is unchanged unless a property of the container is changed. This valuecan be used to implement optimistic concurrency with the blob containers.
LastModified This property indicates when the container was last modified.
PublicAccess This property indicates the level of public access that is allowed on the container. Valid values include Blob, Container, Off, andUnknown.
HasImmutabilityPolicy This property indicates whether the container has an immutability policy. An immutability policy will help ensure that blobs arestored for a minimum amount of retention time.
HasLegalHold This property indicates whether the container has an active legal hold. A legal hold will help ensure that blobs remainunchanged until the hold is removed.

Manipulating blob container metadata in .NET

Using the existing CloudBlobContainer variable (named container), you can set and retrieve custom metadata for the container instance. This metadata is hydrated when you call the FetchAttributes or FetchAttributesAsync method on your blobor container to populate the Metadata collection.

The following code example sets metadata on a container. In this example, we use the collection’s Add method to set a metadata value:

container.Metadata.Add("docType", "textDocuments");

In the next example, we set the metadata value by using implicit key/value syntax:

container.Metadata["category"] = "guidance";

To persist the newly set metadata, you must call the SetMetadataAsync method of the CloudBlobContainer class:

await container.SetMetadataAsync();

 

The Lease Blob operation

The Lease Blob operation establishes and manages a lock on a blob for write and delete operations. The lock duration can be 15 to 60 seconds, or can be infinite. In versions prior to 2012-02-12, the lock duration is 60 seconds.

Important: Starting in version 2012-02-12, some behaviors of the Lease Blob operation differ from previous versions. For example, in previous versions of the Lease Blob operation you could renew a lease after releasing it. Starting in version 2012-02-12, this lease request will fail, while calls using older versions of Lease Blob still succeed.

The Lease Blob operation can be called in one of five modes:

  • Acquire, to request a new lease.
  • Renew, to renew an existing lease.
  • Change, to change the ID of an existing lease.
  • Release, to free the lease if it is no longer needed so that another client may immediately acquire a lease against the blob.
  • Break, to end the lease but ensure that another client cannot acquire a new lease until the current lease period has expired.

Request

The Lease Blob request may be constructed as follows. HTTPS is recommended. Replace myaccount with the name of your storage account:

PUT Method Request URI HTTP Version
https://myaccount.blob.core.windows.net/mycontainer/myblob?comp=lease HTTP/1.1
Emulated Storage Service URI

When making a request against the emulated storage service, specify the emulator hostname and Blob service port as 127.0.0.1:10000, followed by the emulated storage account name:

PUT Method Request URI HTTP Version
http://127.0.0.1:10000/devstoreaccount1/mycontainer/myblob?comp=lease HTTP/1.0

HTTP/1.1

URI Parameters

The following additional parameters may be specified on the request URI.

Parameter Description
timeout Optional. The timeout parameter is expressed in seconds.
Request Headers

The following table describes required and optional request headers.

Request Header Description
Authorization Required. Specifies the authentication scheme, account name, and signature.
Date or x-ms-date Required. Specifies the Coordinated Universal Time (UTC) for the request.
x-ms-version Optional. Specifies the version of the operation to use for this request.
x-ms-lease-id: <ID> Required to renew, change, or release the lease.

The value of x-ms-lease-id can be specified in any valid GUID string format.

x-ms-lease-action: <acquire ¦ renew ¦ change ¦ release ¦ break> acquire: Requests a new lease. If the blob does not have an active lease, the Blob service creates a lease on the blob and returns a new lease ID. If the blob has an active lease, you can only request a new lease using the active lease ID, but you can specify a new x-ms-lease-duration, including negative one (-1) for a lease that never expires.

renew: Renews the lease. The lease can be renewed if the lease ID specified on the request matches that associated with the blob. Note that the lease may be renewed even if it has expired as long as the blob has not been modified or leased again since the expiration of that lease. When you renew a lease, the lease duration clock resets.

change: Version 2012-02-12 and newer. Changes the lease ID of an active lease. A change must include the current lease ID in x-ms-lease-id and a new lease ID in x-ms-proposed-lease-id.

release: Releases the lease. The lease may be released if the lease ID specified on the request matches that associated with the blob. Releasing the lease allows another client to immediately acquire the lease for the blob as soon as the release is complete.

break: Breaks the lease, if the blob has an active lease. Once a lease is broken, it cannot be renewed. Any authorized request can break the lease; the request is not required to specify a matching lease ID. When a lease is broken, the lease break period is allowed to elapse, during which time no lease operation except break and release can be performed on the blob. When a lease is successfully broken, the response indicates the interval in seconds until a new lease can be acquired.

A lease that has been broken can also be released, in which case another client may immediately acquire the lease on the blob.

x-ms-lease-break-period: N Version 2012-02-12 and newer, optional. For a break operation, this is the proposed duration of seconds that the lease should continue before it is broken, between 0 and 60 seconds. This break period is only used if it is shorter than the time remaining on the lease. If longer, the time remaining on the lease is used. A new lease will not be available before the break period has expired, but the lease may be held for longer than the break period. If this header does not appear with a break operation, a fixed-duration lease breaks after the remaining lease period elapses, and an infinite lease breaks immediately.
x-ms-lease-duration: -1 ¦ N Version 2012-02-12 and newer, only allowed and required on an acquire operation. Specifies the duration of the lease, in seconds, or negative one (-1) for a lease that never expires. A non-infinite lease can be between 15 and 60 seconds. A lease duration cannot be changed using renew or change.
x-ms-proposed-lease-id: <ID> Version 2012-02-12 and newer, optional for acquire, required for change. Proposed lease ID, in a GUID string format.The Blob service returns 400 (Invalid request) if the proposed lease ID is not in the correct format.
Origin Optional. Specifies the origin from which the request is issued. The presence of this header results in cross-origin resource sharing headers on the response.
x-ms-client-request-id Optional. Provides a client-generated, opaque value with a 1 KB character limit that is recorded in the analytics logs when storage analytics logging is enabled. Using this header is highly recommended for correlating client-side activities with requests received by the server.
Request Body

None.

Sample Request

The following sample request shows how to acquire a lease:

Request Syntax: PUT https://myaccount.blob.core.windows.net/mycontainer/myblob?comp=lease HTTP/1.1 Request Headers: x-ms-version: 2015-02-21 x-ms-lease-action: acquire x-ms-lease-duration: -1 x-ms-proposed-lease-id: 1f812371-a41d-49e6-b123-f4b542e851c5 x-ms-date: <date> Authorization: SharedKey testaccount1:esSKMOYdK4o+nGTuTyeOLBI+xqnqi6aBmiW4XI699+o=

Response

The response includes an HTTP status code and a set of response headers.

Status Code

The success status codes returned for lease operations are the following:

  • Acquire: A successful operation returns status code 201 (Created).
  • Renew: A successful operation returns status code 200 (OK).
  • Change: A successful operation returns status code 200 (OK).
  • Release: A successful operation returns status code 200 (OK).
  • Break: A successful operation returns status code 202 (Accepted).
Response Headers

The response for this operation includes the following headers. The response may also include additional standard HTTP headers. All standard headers conform to the HTTP/1.1 protocol specification.

Syntax Description
ETag The ETag header contains a value that you can use to perform operations conditionally.

This header is returned for requests made against version 2013-08-15 and later, and the ETag value will be in quotes.

The Lease Blob operation does not modify this property.

Last-Modified The date/time that the blob was last modified. The date format follows RFC 1123.

Any write operation on the blob, including updates on the blob’s metadata or properties, changes the last-modified time of the blob. The Lease Blob operation does not modify this property.

x-ms-lease-id: <id> When you request a lease, the Blob service returns a unique lease ID. While the lease is active, you must include the lease ID with any request to write to the blob, or to renew, change, or release the lease.

A successful renew operation also returns the lease ID for the active lease.

x-ms-lease-time: seconds Approximate time remaining in the lease period, in seconds. This header is returned only for a successful request to break the lease. If the break is immediate, 0 is returned.
x-ms-request-id This header uniquely identifies the request that was made and can be used for troubleshooting the request.
x-ms-version Indicates the version of the Blob service used to execute the request. This header is returned for requests made against version2009-09-19 and later.
Date A UTC date/time value generated by the service that indicates the time at which the response was initiated.
Access-Control-Allow-Origin Returned if the request includes an Origin header and CORS is enabled with a matching rule. This header returns the value of the origin request header in case of a match.
Access-Control-Expose-Headers Returned if the request includes an Origin header and CORS is enabled with a matching rule. Returns the list of response headers that are to be exposed to the client or issuer of the request.
Access-Control-Allow-Credentials Returned if the request includes an Origin header and CORS is enabled with a matching rule that does not allow all origins.This header will be set to true.

Authorization

This operation can be called by the account owner and by any client with a shared access signature that has permission to write to this blob or its container.

 

Move items in Blob storage by using AzCopy

Like with Azure Files, you can use AzCopy to copy blobs between storage containers. The basic syntax for AzCopy commands is:

AzCopy /Source:<source> /Dest:<destination> [Options]

Downloading blobs

You can download blobs from a container to your local computer by using the following command:

AzCopy /Source:https://myaccount.blob.core.windows.net/mycontainer /Dest:C:\myfolder /SourceKey:key /Pattern:”abc.txt”

Alternatively, you can remove the /Pattern option to download all the blobs from the container:

AzCopy /Source:https://myaccount.blob.core.windows.net/mycontainer /Dest:C:\myfolder /SourceKey:key /S

Remember, blobs do not have a folder hierarchy. The entire path of the blob constitutes the name. Assuming you are running the download command in the C:\myfolder path and are downloading a blob named vd1\a.txt, the AzCopy tool will hydrate the blob inthe following local path:

C:\myfolder\vd1\a.txt

You can also perform pattern matching before downloading blobs. In this example, all blobs beginning with the prefix “a” are downloaded:

AzCopy /Source:https://myaccount.blob.core.windows.net/mycontainer /Dest:C:\myfolder /SourceKey:key /Pattern:a /S

Copying blobs between containers

The simplest example of copying blobs would be to copy blobs from one container in the storage account to another:

AzCopy /Source:https://myaccount.blob.core.windows.net/mycontainer1 /Dest:https://myaccount.blob.core.windows.net/mycontainer2 /SourceKey:key /DestKey:key /Pattern:abc.txt

Because you used the full URI to copy blobs within a storage account, you can copy blobs to another storage account using the destination URI. The only difference in this example is the base of the URI referencing a different storage account:

AzCopy /Source:https://sourceaccount.blob.core.windows.net/mycontainer1 /Dest:https://destaccount.blob.core.windows.net/mycontainer2 /SourceKey:key1 /DestKey:key2 /Pattern:abc.txt

By default, AzCopy copies data between two storage endpoints asynchronously. Therefore, the copy operation runs in the background by using spare bandwidth capacity that has no Service Level Agreement (SLA) in terms of how fast a blob is copied, and AzCopyperiodically checks the copy status until the copying has completed or failed.

The /SyncCopy option ensures that the copy operation gets a consistent speed. AzCopy performs the synchronous copy by downloading the blobs to copy from the specified source to local memory and then uploading them to the Blob storage destination:

AzCopy /Source:https://myaccount1.blob.core.windows.net/myContainer/ /Dest:https://myaccount2.blob.core.windows.net/myContainer/ /SourceKey:key1 /DestKey:key2 /Pattern:ab /SyncCopy

Copying blobs to Azure Files shares

Using your knowledge of AzCopy to copy blobs and file shares, you can use the same commands to copy between storage types. The only things you would need to change are the URIs used in the /Source and /Dest options. In this first example, we copy contentfrom an Azure Files share to a blob:

AzCopy /Source:https://myaccount1.file.core.windows.net/myfileshare/ /Dest:https://myaccount2.blob.core.windows.net/mycontainer/ /SourceKey:key1 /DestKey:key2 /S

In this next example, we copy content from a blob to an Azure Files share:

AzCopy /Source:https://myaccount1.blob.core.windows.net/mycontainer/ /Dest:https://myaccount2.file.core.windows.net/myfileshare/ /SourceKey:key1 /DestKey:key2 /S