In part I of this blog post, we started comparing Windows Azure Blob Storage and Amazon Simple Storage Service (S3). We covered basic concepts and compared pricing and features of Blob Container and Buckets. You can read that blog post here: https://gauravmantri.com/2012/05/09/comparing-windows-azure-blob-storage-and-amazon-simple-storage-service-s3part-i/
In this post, we’re going to compare Blobs and Objects from features point of view.
Like the previous post, we’re going to refer Windows Azure Blob Storage as WABS and Amazon Simple Storage Service as AS3 in the rest of this blog post for the sake of brevity.
Concepts
Before we talk about these two services in greater detail, I think it is important to get some concepts clear about blobs and objects. This section is taken verbatim from the previous post.
Blobs and Objects: Simply put blobs (in WABS) and objects (in AS3) are the files in your cloud file system. They go into blob containers and buckets respectively.
A few comments about blobs and objects:
- There is no limit on the number of blobs and objects you can store. While AS3 does not tell you the maximum storage capacity allocated for you, the total number of blobs in WABS is restricted by the size of your storage account (100 TB currently).
- The maximum size of an object you can store in AS3 is 5 TB where as the maximum size of a blob in WABS is 1 TB.
- In WABS, there are two kinds of blobs – Block Blobs and Page Blobs. Block Blobs are suitable for streaming payload (e.g images, videos, documents etc.) and can be of a maximum of 200 GB in size. Page Blobs are suitable for random read/write payload and can be of a maximum of 1 TB in size. A common use case of a page blob is a VHD mounted as a drive in a Windows Azure role. In AS3, there is no such distinction. To learn more about block blobs and page blobs, click here.
- Both systems are quite feature rich as far as operations on blobs and objects are concerned. You can copy, upload, download and perform other operations on them.
- While both systems allow you to protect your content from unauthorized access, the ACL mechanism is much granular in AS3 where you can set custom ACL on each object in a bucket. In WABS, it is at a blob container level.
Two of the most important functions that you would work with are uploading and downloading. Let’s take a moment and understand them in somewhat greater detail and then we’ll compare the functionality offered.
Uploading Blobs and Objects
Let’s talk about how you can upload blobs and objects in a blob container or bucket respectively. There are two mechanisms by which you can perform an upload. You can either upload an entire blob or object in a single attempt or you can split them in chunks (called blocks or pages in WABS and parts in AS3).
Uploading in Single Attempt
Let’s say the data you’re uploading is small and you have relatively good Internet connectivity, you can upload that entire data in a single attempt. In WABS you would use Put Blob functionality. In AS3 you would use Put Object.
Uploading in Chunks
As mentioned above, you could store really large data as blobs and objects in WABS and AS3 respectively. If you’re trying to upload really large data, it becomes impractical to try and upload the entire data in single attempt. Luckily both WABS and AS3 give you the ability to split the data in smaller chunks (called blocks or pages in WABS and parts in AS3) and then upload these chunks. To do so in WABS, for block blobs you would use Put Block and Put Block List functionality and for page blobs you would use Put Page functionality. To do so in AS3, you would use Initiate Multipart Upload, Upload Part and either Complete Multipart Upload or Abort Multipart Upload functionality.
There are many reasons why you should consider uploading data in chunks:
- You’re trying to upload really large data. Please note that in WABS you could have a single block blob of up to 200 GB and a single page blob of 1 TB and in AS3 you could have a single object up to 5 TB in size. Thus it becomes impractical to upload data of such size in a single go.
- Your Internet connectivity may be poor.
- Both WABS and AS3 are cloud services designed to cater to hundreds of thousands of concurrent users. They will timeout your requests if they find out your requests are taking longer than allowed limit. For example, WABS has a threshold where they allow your requests 10 minutes to upload 1 MB of data. If you fail to upload this much data in 10 minutes, WABS will time you out.
- Splitting large data in chunks allows you to perform parallel uploads thus you would be able to upload data much faster.
- In case a chunk fails, you could just retry just that chunk. However if you’re uploading data in a single go and that fails, you would need to do entire upload again which is not very efficient.
- You’re restricted by the system. WABS actually prohibits you from uploading data in a single go if the size of the data is more than 64 MB i.e. if you’re data size is more than 64 MB, you must upload it by splitting in blocks or pages. While AS3 does not have any restriction, they also recommend to use multipart uploads if your object size is more than 100 MB.
Now let’s see how uploading in chunks work in each system. Let’s say you want to upload a 100 MB file in each system and you want upload it in chunks.
WABS
Let’s say size of each chunk is 1 MB each (though it is not really required that all your chunks are of same size). Essentially what you need to do is upload 100 chunks. For the sake of explaining, let’s assume that you’re uploading a block blob in WABS instead of a page blob. In WABS, for each block (chunk) you will assign a unique identifier (called BlockId) and start uploading that block using Put Block function. BlockId is a Base64 encoded string value maximum length of which could be 64 bytes. Also all block ids (100 in our case) must be of same length. It doesn’t matter in which order you upload the blocks. You can upload blocks in parallel if you would like. Once a block is uploaded successfully, WABS store it somewhere in its infrastructure. WABS will keep these blocks for a period of 7 days. Once all block are uploaded successfully, then you would call Put Block List functionality to commit these blocks. Until the time you call “Put Block List”, you will not be able to access the blob. If you don’t commit the blocks in 7 days, they will be garbage collected by the system. When you call Put Block List, based on the order of BlockIds passed in its request payload, WABS construct the blob and then make it accessible. While it is not important how you name the BlockIds (all of them can be GUIDs), it is really important to send the BlockId list in proper order to Put Block List functionality.
There are a few restrictions here:
- A blob can only be split into a maximum of 50000 blocks.
- A blob can have a maximum of 100000 uncommitted blocks at any given time.
- The set of uncommitted blocks cannot exceed 400 GB in total size.
- As mentioned above, all BlockIds in a blob are of same length. It is not acceptable to have BlockIds like block8, block9, block11.
- Maximum length of a BlockId can be 64 bytes.
AS3
Let’s say the size of each chunk is 5 MB. In AS3, again for each part (chunk) you will assign a unique identifier (called Part Number). A part number uniquely identifies a part and also defines its position within the object being created. A part number can be any number between 1 and 10000 however AS3 requires them to be sorted i.e. in our example 0 – 5 MB part must have “1” as part number and 5 – 10 MB part must have "2" as part number. However before you start uploading these parts, first you would need to call Initiate Multipart Upload function which will tell AS3 that you have started uploading an object in parts. This function would return you an “Upload Id”. Next you would call Upload Part function for each of the parts. You would need to provide both “Upload Id” and “Part Number” for each part. It doesn’t matter in which order you upload the parts. You can upload parts in parallel if you would like. Once a part is uploaded successfully, AS3 store it somewhere in its infrastructure. Once all parts have been uploaded, you would call Complete Multipart Upload function to commit these parts. Till the time you call this function, object will not be accessible. If you wish to cancel the process, you could call Abort Multipart Upload functionality anytime during this process.
There are a few restrictions here as well:
- Since the documentation says that a part number can be any number between 1 and 10000, I am assuming that an object can be split into a maximum of 10000 parts.
- Since the part number defines the position of the part within the object being created, it’s value becomes important.
- Each part must be at least 5 MB in size except for the last part where there is no size restriction.
Downloading Blobs and Objects
Now let’s talk about how you can download blobs and objects from a blob container or bucket respectively. There are two mechanisms by which you can perform an download. You can either download an entire blob or object in a single attempt or you can split them in chunks.
Unlike uploading where there are many functions to perform upload (Put Blob, Put Block, Put Block List in WABS and Put Object, Initiate Multipart Upload, Upload Part, Complete Multipart Upload in AS3), there is only a single function to perform a download. That function is Get Blob in WABS and Get Object in AS3.
Downloading in Single Attempt
If the data to be downloaded is small and you have good Internet connectivity, you can download that entire data in single attempt using Get Blob in WABS and Get Object in AS3.
Downloading in Chunks
If the data to be downloaded is large and you think you will not be able to download it completely in a single attempt, you can download the data in chunks. You will still use the same functions as above however you would use an additional request header called “Range” and specify the byte range you would want to download.
The reasons for uploading data in chunks also apply here as to why you would want to download data in chunks.
This is the process you would typically follow:
- First you would need to know the size of the blob or object you wish to download. Let’s say you wish to download a 100 MB item.
- Next you determine the chunk size based on your requirement. Let’s say you’re comfortable with 1 MB chunks.
- Now you will repeatedly call Get Blob or Get Object functionality and pass appropriate values in Range request header. So if you’re downloading sequentially, your first request will have the value of this header as “0 – 1048575” (0 – 1 MB) and your second request will have the value of this header as “1048576 – 2097151” (1 – 2 MB) and so on and so forth.
- Once each chunk is downloaded, you will store that data somewhere.
- Once every chunk is downloaded, you would create an empty file of 100 MB size and start fill that file with all the chunks you have downloaded.
Function/Feature Summary
Following table summarizes the list of functions provided by WATS and ASDB.
Now we’ll explore these functions in somewhat more details.
Put Blob/PUT Object
| WABS | AS3 | |
| Put Blob/PUT Object | Yes | Yes |
As explained earlier, this function adds a block or a page blob to a blob container in WABS and an object to a bucket in AS3.
A few comments about this functionality in each system:
- In both systems, this function will overwrite an existing item with the same name. In AS3, if versioning is enabled on the bucket in which you’re uploading, a copy of the object will be created. In WABS, you would need to handle the versioning on your own by calling Snapshot Blob function.
- Both systems allow you to define some properties on items being uploaded like cache control, content type, etc.
- Both systems allow you to send MD5 hash of the content to check for data integrity between sender and receiver.
- You can also set ACL on an object while creating it in AS3 which you can’t do in WABS.
- As mentioned in the previous post, AS3 allows you to store objects with reduced redundancy. When you call this function, you can specify through an optional parameter if you want to store object with reduced redundancy. If this parameter is omitted, the object is saved with standard redundancy.
- Both systems allow you to set custom metadata on blobs and objects in form of a collection of name/value pairs. In both systems, maximum size of metadata can be 8 KB.
- When you create a Page Blob using this functionality in WABS, you only initiate the page blob. You don’t put data in that blob. To insert data in a page blob you would use Put Page functionality.
- When you create a Block Blob in WABS or an object in AS3, the data is sent in the request payload.
- Maximum size of block blob which can be created with this functionality is 64 MB. If the size of the blob is more than 64 MB, it must be split into blocks and uploaded using Put Block and Put Block List functionality.
- WABS allow you to specify preconditions which must be satisfied for successful completion of this operation by means of conditional headers (If-Modified-Since, If-Unmodified-Since, If-Match, If-None-Match).
- Since AS3 supports server-side encryption of objects, you can specify if you want the object to be encrypted by AS3 while making this request. WABS does not support server-side encryption. You would need to encrypt the data yourself before calling this function.
POST Object
| WABS | AS3 | |
| POST Object | No | Yes |
This function adds an object to a specified bucket using HTML forms. POST is an alternate form of PUT that enables browser-based uploads as a way of putting objects in buckets. Parameters that are passed to PUT via HTTP Headers are instead passed as form fields to POST in the multipart/form-data encoded message body.
Get Blob/GET Object
| WABS | AS3 | |
| Get Blob/GET Object | Yes | Yes |
As explained earlier, this function downloads a blob from a blob container in WABS and an object from a bucket in AS3.
A few comments about this functionality in each system:
- As mentioned above, you can do a partial download by specifying the bytes you want to download in the “Range” request header.
- In WABS, this function also returns the custom metadata for the blob being downloaded.
- Both systems allow you to specify preconditions which must be satisfied for successful completion of this operation by means of conditional headers (If-Modified-Since, If-Unmodified-Since, If-Match, If-None-Match).
- One neat feature in AS3 is that you can override values for certain response headers using query parameters. The catch is that you can’t do this with anonymous requests. The request would need to be signed either using an Authorization header or using a pre-signed URL.
- You can also use this functionality to get versioned blobs and objects as well. In order to get versioned blob you would need to specify blob’s snapshot date/time in WABS and version id in AS3 to get a specific version of an object. If these parameters are omitted, current version is returned.
GET Object torrent
| WABS | AS3 | |
| GET Object torrent | No | Yes |
As mentioned in the previous post, AS3 allows you to serve objects through BitTorrent protocol also which can save you bandwidth when distributing data to large number of users. This function is used for that purpose. To learn more about using BitTorrent with AS3, click here.
You can only invoke this function only on objects which are less than 5 GB in size.
GET Object ACL
| WABS | AS3 | |
| GET Object ACL | No | Yes |
As mentioned above, you can specify ACL at individual object level in AS3 (you can only define ACL at blob container level in WABS). This function is used to fetch information about ACL set on an object.
One interesting thing here is that if you have different versions of an object, each version can have different ACL. To get ACL for a particular version of an object, you would need to pass the id of that version in your request.
PUT Object ACL
| WABS | AS3 | |
| PUT Object ACL | No | Yes |
This function allows you to set ACL on an object. As mentioned above, each version of an object can have different ACL. To set ACL for a particular version of an object, you would need to pass the id of that version in your request.
Get Blob Properties/HEAD Object
| WABS | AS3 | |
| Get Blob Properties/HEAD Object | Yes | Yes |
This function is used to fetch properties of a blob and custom metadata of an object. It does not return the contents of a blob.
A few comments about this functionality in each system:
- Get Blob Properties in WABS returns all user-defined metadata, standard HTTP properties, and system properties for the blob where as HEAD Object in AS3 only returns metadata for the object.
- Both systems allow you to specify preconditions which must be satisfied for successful completion of this operation by means of conditional headers (If-Modified-Since, If-Unmodified-Since, If-Match, If-None-Match).
- You can also use this functionality to get properties for a particular version of a blob or an object. In order to get that information, you would need to specify blob’s snapshot date/time in WABS and version id in AS3 to get a specific version. If these parameters are omitted, information about most current version is returned.
Set Blob Properties
| WABS | AS3 | |
| Set Blob Properties | Yes | No |
This function sets system properties on a blob in WABS. This function is not available in AS3. You could only set the system properties when an object is created.
The properties you could set with this operation are: Cache Control, Content Type, Content MD5, Content Encoding, and Content Language. Please note that you can’t change these properties for a snapshot of a blob.
WABS allow you to specify preconditions which must be satisfied for successful completion of this operation by means of conditional headers (If-Modified-Since, If-Unmodified-Since, If-Match, If-None-Match).
Get Blob Metadata/Head Object
| WABS | AS3 | |
| Get Blob Metadata/Head Object | Yes | Yes |
This function returns all user-defined metadata for the specified blob or object. You can also use this functionality to get metadata for a particular version of a blobs or an object. In order to get that information, you would need to specify blob’s snapshot date/time in WABS and version id in AS3 to get a specific version of an object. If these parameters are omitted, information about most current version is returned.
Set Blob Metadata
| WABS | AS3 | |
| Set Blob Metadata | Yes | No |
This function allows you to set user-defined metadata for the specified blob as one or more name-value pairs.
Like with setting the metadata for a blob container, here also:
- This function overwrites the existing metadata thus it is not possible to update just one name/value pair.
- The maximum size of metadata can be 8 KB.
- The metadata name must be a valid C# identifier.
Delete Blob/DELETE Object
| WABS | AS3 | |
| Delete Blob/DELETE Object | Yes | Yes |
This function allows you to delete a blob or an object from storage.
A few comments about this functionality in each system:
- In AS3, this function removes the null version (if there is one) of an object and inserts a delete marker, which becomes the latest version of the object. If there isn’t a null version, AS3 does not remove any objects.
- You can use this function to remove only the snapshots of a blob in WABS without deleting the base blob. If base blob is deleted, all of its snapshots are deleted as well.
- Based on my understanding, in AS3 if you delete an object, its version still remain available which I think is a very cool feature to safeguard against accidental deletes.
- You can also use this functionality to delete a particular version of a blob or an object. In order to do that, you would need to specify blob’s snapshot date/time in WABS and version id in AS3 to delete that specific version.
- WABS allow you to specify preconditions which must be satisfied for successful completion of this operation by means of conditional headers (If-Modified-Since, If-Unmodified-Since, If-Match, If-None-Match).
- One really cool feature in AS3 is Multi Factor Authentication (MFA) Delete. This provides an additional layer of security. To learn more about MFA Delete, click here.
Delete Multiple Objects
| WABS | AS3 | |
| Delete Multiple Objects | No | Yes |
This is a cool feature in AS3 which allows you to delete multiple objects from a bucket using a single HTTP request instead of sending delete request for each object. For this operation you would need to know the keys of the objects you want to delete. You can also delete versions of one or more objects as well.
In a single request, you can delete up to 1000 objects.
Copy Blob/Put Object – Copy
| WABS | AS3 | |
| Copy Blob/Put Object – Copy | Yes | Yes |
As the name suggests, this function copies a blob from a source location to a target location.
A few comments about this functionality in each system:
- Both systems allow you to specify preconditions which must be satisfied for successful completion of this operation by means of conditional headers (If-Modified-Since, If-Unmodified-Since, If-Match, If-None-Match). You can specify these conditional headers on both source and target items in WABS and on source item in AS3.
- WABS only allow you to copy item from one blob container to another blob container in same storage account. AS3 does not have this restriction. As long as the source and target buckets belong to the same account, an object can be copied. However if you create an object using the multipart upload APIs, you cannot copy the object between regions.
- Both systems allow you to either copy the existing metadata or define new metadata for the target item.
- Internally AS3 implements this functionality as a combination of GET Object and PUT Object.
- Using this functionality you can only perform this operation on objects which are up to 5 GB in size. If an object is more than 5 GB in size, you would need to perform a multipart upload.
- In AS3, when copying an object it’s ACL is not preserved and is set private by default.
Cool Tips:
- Neither of the system support “Move” functionality. However you can perform a “move” operation by first copying item from source to target and then deleting the source.
- You can also “promote” a version of a blob or an object to the most current version. To do so you would pass the versioned blob (by specifying its snapshot) or object (by specifying its version id) as source and the target as the unversioned blob or object.
Snapshot Blob
| WABS | AS3 | |
| Snapshot Blob | Yes | No |
This function creates a read-only copy of a blob. This is how versioning is implemented in WABS. As I mentioned in my previous post, version in AS3 is implemented at bucket level and once versioning is enabled, AS3 is responsible for creating different versions of an object. In WABS, versioning is implemented at individual blob level and it is your responsibility to create different versions of a blob. This function facilitates that.
A few comments about this functionality:
- This function creates a read-only copy of a blob. A blob’s snapshot can be read, copied or deleted but it can never be modified.
- Unlike AS3 where a version id is assigned to a version of an object, here a date/time value is assigned to uniquely identify a snapshot.
- WABS allow you to specify preconditions which must be satisfied for successful completion of this operation by means of conditional headers (If-Modified-Since, If-Unmodified-Since, If-Match, If-None-Match).
- You can promote a blob snapshot using Copy Blob operation. Promoting a snapshot means replacing the base blob with one of its snapshot.
Lease Blob
| WABS | AS3 | |
| Lease Blob | Yes | No |
This function lets you acquire an exclusive one minute lock on a blob so that no other caller can modify that blob. This function is really handy when you have multiple “workers” trying to perform same operation and you want only one of them to perform the work. Steve Marx has written a nice blog post explaining the utility of this function. You can read that post here: http://blog.smarx.com/posts/managing-concurrency-in-windows-azure-with-leases.
Initiate Multipart Upload
| WABS | AS3 | |
| Initiate Multipart Upload | No | Yes |
As explained earlier in this post, this function is used to start uploading data in parts in AS3. Please see “Uploading Blobs and Objects” section for more details.
Put Block/Upload Part
| WABS | AS3 | |
| Put Block/Upload Part | Yes | Yes |
As explained earlier in this post, this function is used to upload chunks (block in case of a block blob in WABS and part in case of an object in AS3) of data. Please see “Uploading Blobs and Objects” section for more details.
Put Block List/Complete Multipart Upload
| WABS | AS3 | |
| Put Block List/Complete Multipart Upload | Yes | Yes |
As explained earlier in this post, this function is used to commits a blob or an object in respective storage. You call this function when you have uploaded all blocks or parts. Please see “Uploading Blobs and Objects” section for more details.
Get Block List/List Parts
| WABS | AS3 | |
| Get Block List/List Parts | Yes | Yes |
This function returns the list of blocks or parts that have been uploaded as part of a block blob or an object using multipart upload.
A few comments about this functionality in each system:
- WABS maintain two block lists for a blob: Committed Block List which is the list of blocks that have been successfully committed to a given blob and Uncommitted Block List which is list of blocks that have been uploaded for a blob using Put Block, but that have not yet been committed.
- In a single request to this function, the list AS3 returns can contain a maximum of 1000 part numbers. If there are more parts, then AS3 returns a continuation token using which next set of part numbers can be fetched. You can instruct AS3 to return a list which contains less than 1000 part numbers as well.
- WABS allow you to specify preconditions which must be satisfied for successful completion of this operation by means of conditional headers (If-Modified-Since, If-Unmodified-Since, If-Match, If-None-Match).
Abort Multipart Upload
| WABS | AS3 | |
| Abort Multipart Upload | No | Yes |
As explained earlier in this post, this function is used to cancel an existing multipart upload process in AS3. Please see “Uploading Blobs and Objects” section for more details.
The way block upload functionality is implemented in WABS, you really don’t need this functionality. In WABS, you never abort an upload process. If you started uploading a blob by splitting it into blocks and you don’t call “Put Block List” functionality in 7 days, uploaded blocks are garbage collected by WABS.
Upload Part – Copy
| WABS | AS3 | |
| Upload Part – Copy | No | Yes |
As mentioned in Copy Blob/Put Object – Copy section, in AS3 if the object you’re trying to copy is more than 5 GB in size, you can’t use Put Object – Copy function. You would need to copy it using Multipart Upload functionality. This function is used for that purpose only. The difference between this function and Upload Part is that here instead of passing the data in the request body, you specify the source object in source_bucket/source_object format and byte range that you wish to copy. You would need to do this for all the parts of your source object. Before you call this function, you would need to call “Initiate Multipart Upload” which will give you an upload id which needs to be provided to this function. To complete the copy function, you would either need to call “Complete Multipart Upload” or “Abort Multipart Upload”.
Put Page
| WABS | AS3 | |
| Put Page | Yes | No |
When you create a page blob using Put Blob functionality, you just initiate that blob i.e. you create an empty page blob with no bytes in it. With that function, you never provide the data that will go in that blob. To insert/update data in a page blob, you use this function. Unlike block blobs where you would need to call Put Block List functionality to commit the data, as soon as you successfully executes this function, data is committed in the blob.
A few comments about this function:
- In order to put data, you would need to specify “Byte Range” in your request header. The start offset of this range must be a modulus of 512 and the end offset must be a modulus of 512 – 1. Examples of valid byte ranges are 0-511, 512-1023, etc.
- Using this function, you can either write some data into the page blob or clear a particular byte range.
- WABS allow you to specify preconditions which must be satisfied for successful completion of this operation by means of conditional headers (If-Modified-Since, If-Unmodified-Since, If-Match, If-None-Match).
Get Page Ranges
| WABS | AS3 | |
| Get Page Ranges | Yes | No |
One important distinction between a block blob and a page blob is that for storage you’re charged for the size of a block blob however for a page blob you’re only charged for the number of bytes you’re occupying (read non-zero bytes) in that page blob. To give you an example, let’s say you create a 2 GB block blob and a 2 GB page blob and both of them are filled with 0 bytes. In case of block blob you will be charged for entire 2 GB however in case of page blob you will not be charged anything because your blob is empty. Now let’s say that you write 1024 bytes using Put Page functionality in that page blob. You will now be charged only for 1024 bytes of storage.
To get the list of occupied pages, you call this function. This function returns you a list of non-overlapping valid page ranges, sorted by increasing address page range.
There is another advantage of using this function. When you’re downloading a page blob, if you call this function you will know exactly where your data is contained in that blob. You could then only download those ranges by specifying them in “Range” request header in Get Blob functionality.
Summary
As we saw in this post, both systems offer similar feature set. There are some features which are present in one system which we wish for in another system but generally speaking the disparity between them is not so much like blob containers and buckets.
A few comments (Disclaimers)
- It is not the intent of the blog post to prove that one service is superior over other. I just wanted to have a very objective comparison of these two services.
- I make my living using Windows Azure Platform (if you wish, you can call me Windows Azure fan boy
). This does not mean that I have negative things to say about Amazon Cloud Computing Platform. It’s just that I never had a chance to play with Amazon just yet. - Since I haven’t played with Amazon platform just yet, the information I have presented in this blog post about Amazon S3 is solely based on my understanding of this service based on the documentation. Very likely, I may be wrong about some of things I have written here. Please note that it is not done intentionally and you can blame my lack of knowledge for that. If you find such issues, please let me know and I will fix them ASAP.
