In this blog post, we’re going to compare Windows Azure Queue Service and Amazon Simple Queue Service from core functionality point of view.
For the sake of brevity, we’re going to refer Windows Azure Queue Service as WAQS and Amazon Simple Queue Service as ASQS in the rest of this blog post.
From fundamental functionality point of view, both WAQS and ASQS provides similar functionality. Essentially both of them can be used to facilitate communication between disconnected systems. Here’s how both define this service:
WAQS: Queues are used for reliable, persistent messaging between applications. You can use Queues to transfer messages between applications or services in Windows Azure.1
ASQS: Amazon Simple Queue Service (Amazon SQS) offers a reliable, highly scalable, hosted queue for storing messages as they travel between computers. By using Amazon SQS, developers can simply move data between distributed components of their applications that perform different tasks, without losing messages or requiring each component to be always available. Amazon SQS makes it easy to build an automated workflow, working in close conjunction with the Amazon Elastic Compute Cloud (Amazon EC2) and the other AWS infrastructure web services.2
In both systems, you can create one or more queues which will hold zero or more messages to process. Common use case scenario includes one or more publishers (a process or an application) pushing the messages into queues and one or more subscribers frequently polling the queues for messages for processing.
Both systems provide a REST based API for working with queues and messages and other higher level language libraries which are essentially wrappers implementing REST API. Over the years, both systems have evolved in terms of functionality provided. In both systems, each release of the API is versioned and is specified as a date. At the time of writing this blog, the service version number for WAQS is 2011-08-18 while that of ASQS is 2011-10-01.
Pricing
Before we talk about the functionality offered in each system, let’s take a moment and talk about the pricing. In both systems, there are no upfront costs. The pricing model is rather simple and consumption based. In both systems, you are charged on usage basis and there are three components:
- Transaction: You’re charged for the for the number of transactions you performed against each system. Simply put, a single transaction can be defined as one call to the functionality in each system. For example, when you create a queue in each system it would be considered as a single transaction. At the time of writing this blog, the pricing for 10000 transactions in both WAQS and ASQS is $0.01 (i.e. $1.00 for a million transactions).
- Storage: You’re charged for the amount of data you store in each system. ASQS does not specify explicitly if you’re charged for storage however WAQS specifies that. At the time of writing this blog, the pricing for storing 1 GB of data in WAQS is $0.125 per month on a daily average basis.
- Data Transfer: You’re charged for the amount of data transferred from/to each of the system. At the time of writing this blog, both systems offer free ingress (i.e. data coming from outside) but charge a fee for data egress (i.e. data going out). Data transferred between ASQS and Amazon EC2 within a single region is free of charge (i.e., $0.00 per GB). Data transferred between ASQS and Amazon EC2 in different regions will be charged at Internet Data Transfer rates on both sides of the transfer. Similarly in WAQS, only the data going out of a data center is charged.
There is also a concept of specialized pricing and both systems offer different pricing packages and offers which you can avail. For more details on pricing, please refer to https://www.windowsazure.com/en-us/pricing/details/ for WAQS and https://aws.amazon.com/sqs/pricing/ for ASQS.
Function/Feature Summary
Following table summarizes the list of functions provided by WAQS and ASQS.
Now we’ll explore these functions in somewhat more details.
Create Queue/CreateQueue
| WAQS | ASQS | |
| Create Queue/CreateQueue | Yes | Yes |
As the name suggests, this function creates a queue in each of the system. Both systems support creating queues. While creating a queue in WAQS, you can define custom metadata (attributes) for the queue in the form of name/value pair. In some fashion, same thing is available in ASQS also however the attributes in ASQS are system defined and thus fixed. Custom attributes which you can define while creating queues are: VisibilityTimeout, Policy, MaximumMessageSize, MessageRetentionPeriod, and DelaySeconds.
There are a few subtle differences when it comes to naming a queue. Following table summarizes the difference between queue naming conventions in each system:
| WAQS | ASQS | |
| Minimum/Maximum length of queue name | 3/63 | 1/80 |
| Queue name case sensitivity | Lowercase only | Mixed case |
| Valid characters | Alphanumeric, and Dash (-) | Alphanumeric, Dash (-) and Hyphen (_) |
Some other comments about naming queues:
- In WAQS, the first and last letters in the queue name must be alphanumeric. The dash (-) character cannot be the first or last character. Consecutive dash characters are not permitted in the queue name.
- Though I have mentioned that in ASQS, the minimum length of a queue name can be 1 character however no such thing is mentioned in the documentation. Documentation mentions only about the maximum length. Minimum length of 1 character is my assumption. Same is the case with queue name case sensitivity though the example used on the documentation page shows that a queue name can contain mixed case letters.
List Queues/ListQueues
| WAQS | ASQS | |
| List Queues/ListQueues | Yes | Yes |
As the name suggests, this function lists queues in each of the system. In both systems, you can pass a queue name prefix and list of queues returned will start with that prefix. For example, you can pass this prefix as “abc” and both systems will return queues starting with letters “abc” e.g. “abc1”, “abc2” etc.
However there are some differences where the functionality differs. Following table summarizes the differences:
| WAQS | ASQS | |
| Maximum records returned per call | 5000 | 1000 |
| Optionally return metadata | Yes | No |
| Ability to limit the result size | Yes | No |
Some other comments about listing queues:
- In WAQS, if there are more queues available than requested (e.g. desired result size was 100 and there are 1000 queues), the service returns some kind of continuation token which can be used to fetch next set of results. It seems ASQS does not offer that flexibility. It will only return 1000 queues.
- It is not clear from the documentation as to how to fetch all queues in ASQS if there are more than 1000 queues present. One possibility would be to make use of “prefix” functionality however I’m not sure if that would be 100% fool proof. For example, I could repeatedly call ListQueues function and pass all numbers and alphabets (0 – 9, a – z and A – Z) as prefix one at a time. What would happen if I have 1000 queues starting with letters “aa” and then another 1000 queues starting with letter “ab”?
Delete Queue/DeleteQueue
| WAQS | ASQS | |
| Delete Queue/DeleteQueue | Yes | Yes |
As the name suggests, this function deletes a queue in each of the system. The functionality is more or less the same in both systems. Also, even though it may appear that this operation is a synchronous operation, in reality it is not. In both systems, when you send a request to delete a queue the queue is immediately removed from the system (read “marked for deletion”) in the sense that it will not be returned when you try and list the queues however it is sent to a garbage collection process which performs the actual deletion operation.
Some other comments about deleting queue:
- In ASQS, it is recommended that you wait for at least 60 seconds before creating a new queue with the same name but in WAQS no such estimate is provided.
- In WAQS, as soon as service receives the request to delete a queue, that queue becomes unavailable. However based on ASQS documentation, a queue marked for deletion may be accessible for sometime (
Get Queue Metadata/GetQueueAttributes
| WAQS | ASQS | |
| Get Queue Metadata/GetQueueAttributes | Yes | Yes |
As mentioned earlier, both WAQS and ASQS allow you to define additional attributes on a queue. In WAQS, you can define custom attributes (metadata in the form of name/value pair), in ASQS you can only work with system defined properties. “Get Queue Properties” functionality in each system returns you these attributes.
In WAQS, apart from metadata defined for a queue you can also get an approximate number of messages in a queue. Since queues are quite dynamic in nature (your messages may be garbage collected at any time depending on message expiration), WAQS returns you an approximate count. You can use this as a benchmark to see how your queues are performing. Assuming you’re using queues in a scenario where a “publisher” is pushing messages in a queue and one or more “subscribers” pulling messages from a queue to work on and then subsequently delete those messages, a higher number of messages in your queue could mean some issues with your message subscribers.
In ASQS, you get some really interesting properties back. As of the current version (2011-10-01) of ASQS, following properties are returned: VisibilityTimeout, ApproximateNumberOfMessages, ApproximateNumberOfMessagesNotVisible, CreatedTimestamp, LastModifiedTimestamp, QueueArn, MaximumMessageSize, MessageRetentionPeriod, Policy, DelaySeconds, ApproximateNumberOfMessagesDelayed.
From my experience working with Windows Azure, one property I find really interesting here is ApproximateNumberOfMessagesNotVisible. IMHO in the scenario I described above with WAQS, I think this property would come in very handy there.
One difference between getting queue metadata in ASQS and WAQS is that in ASQS you can request to service to return all properties or selected properties (for example just VisibilityTimeout and ApproximateNumberOfMessages) however you don’t have that flexibility in WAQS i.e. in WAQS you can’t request the service to return only selected custom attributes. It will always return all metadata.
Set Queue Metadata/SetQueueAttributes
| WAQS | ASQS | |
| Set Queue Metadata/SetQueueAttributes | Yes | Yes |
Set Queue Properties functionality allows you to set custom metadata/system attributes in each system. In WAQS, using this functionality you can set custom metadata for a queue. In ASQS, using this functionality you can set values for one of the following attributes of a queue: VisibilityTimeout, Policy, MaximumMessageSize, MessageRetentionPeriod, DelaySeconds.
A few interesting things to notice:
- In WAQS, each call to this operation replaces all existing metadata attached to the queue. So if you wish to change value of one key, you would also need to provide name/value pairs for unchanged values as well. If you wish to clear metadata completely, call this operation with no metadata i.e. no key/value pairs.
- In ASQS, in each call to this operation you can only change only one attribute. What this means is that if you wish to change value for two attributes, you would need to call this operation twice. This might have pricing implications as you are charged per transaction as well (howsoever minimal it may be). In this case you would end up paying for two transactions. Based on the current pricing, that would be $0.000002.
- Based on the documentation, furthermore in ASQS each change can take up to 60 seconds to propagate throughout the system whereas in my experience the changes are reflected immediately in WAQS.
GetQueueURL
| WAQS | ASQS | |
| GetQueueURL | No | Yes |
This functionality is only available in ASQS. Simply put, it provides a simple way to retrieve the URL of an SQS queue. This is meaningful in ASQS because ASQS allows you to grant policy (permission) based access to other users (which BTW is a pretty neat feature IMHO) to your queues which you can’t do in WAQS (just yet). In WAQS, all queue/message related functionality is only accessible/invoked by an account owner only.
Put Message/SendMessage
| WAQS | ASQS | |
| Put Message/SendMessage | Yes | Yes |
This functionality can be used to create a new message in a queue in each system. As far as creating a new message is concerned, both systems are quite similar:
- In both systems, maximum size of the message can be 64KB as of the current version at the time of writing this post. If you have a requirement where size of the message can be more than 64KB, general recommendation is that you store the message as a blob in Windows Azure Blob Storage or Amazon Simple Storage Service (S3) and store the blob URL as message body.
- In both systems, you can optionally make a message invisible for a certain amount of time (called VisibilityTimeout in WAQS and DelaySeconds in ASQS). If this parameter is specified, then the message becomes invisible immediately after creation for number of seconds specified in this parameter’s value.
However there are some differences as well:
- In WAQS, you can define when a message will expire automatically. Currently in WAQS, a message can remain in a queue for a maximum of 7 days. After 7 days, that message will be removed from the queue (garbage collected). If you wish to have your message expire sooner than that, you can define it at the time of message creation. Please note that after a message is created, you can’t change the message expiration.
From the documentation it seems that the ability to expire a message is not available in ASQS. It would be your responsibility to delete the message after it has served its purpose. Based on my experience working with Windows Azure, a lot of folks have been asking this feature there. They don’t want system to enforce this maximum 7 day limit.[Edit] A message can remain in the queue for 4 days in ASQS. If a message is not deleted, then the system would delete it automatically. - In WAQS, the initial visibility timeout value cannot be set to a value later than message expiry time i.e. it can have any value more than 0 but less than 7 days. If this parameter is not specified, the message becomes immediately visible to subscribers of this queue. In ASQS, the maximum value that you can assign is 900 (i.e. 15 minutes). Also, if this parameter is omitted in the request then the service would use the value (DelaySeconds) from the parent queue as the default value.
- In WAQS, a message must be in a format that can be included in an XML request with UTF-8 encoding where as in ASQS only the following characters (in Unicode) are allowed: #x9 | #xA | #xD | [#x20 to #xD7FF] | [#xE000 to #xFFFD] | [#x10000 to #x10FFFF].
- If you’re using REST API to create a new message, in ASQS you may need to properly URL encode your message because the message body is passed in URL querystring.
- Based on the documentation, when you create a message in WAQS it always goes to the back of the queue however in ASQS it is not specified whether the message goes to the back of the queue or not.
SendMessageBatch
| WAQS | ASQS | |
| SendMessageBatch | No | Yes |
This functionality allows you create up to 10 messages in a queue in a single request in ASQS. While each message’s maximum size can be 64KB, there is a limitation on total payload size. Maximum payload size (i.e., the sum of all a batch’s individual message lengths) is also 64 KB. There is no equivalent function available in WAQS. In WAQS, if you wish to create multiple messages, you would need to call Put Message functionality multiple times which will have transaction cost implications.
Get Messages/ReceiveMessage
| WAQS | ASQS | |
| Get Messages/ReceiveMessage | Yes | Yes |
This functionality can be used to fetch one or more messages from a queue. Typical use case for fetching messages is that you want one of your subscribers to fetch messages and then work on them. To prevent more than one subscriber fetching same message and working on it, both system supports making messages invisible to other subscribers for some time while subscribers work on those messages. In both WAQS and ASQS, it is achieved by specifying a proper value for “visibilitytimeout” parameter.
Some other comments about this:
- In WAQS, a maximum of 32 messages can be fetched at a time while in ASQS, a maximum of 10 messages can be fetched.
- In WAQS, “visibilitytimeout” parameter is required and should have a value between 1 second and 604800 (7 days) where as in ASQS this parameter is optional. In ASQS, this parameter can have a value between 0 and 43200 (12 hours).
- In WAQS, properties (attributes) about a message returned by the service are fixed. They are: MessageId, InsertionTime, ExpirationTime, PopReceipt, TimeNextVisible, DequeueCount, and MessageText. In ASQS, some of the properties returned by the service are fixed however you can request some additional properties can also be fetched as well. Fixed attributes returned by ASQS are: Body, MD5OfBody, MessageId, and ReceiptHandle. Additional attributes which can be returned by the service are: SenderId, SentTimestamp, ApproximateReceiveCount, and ApproximateFirstReceiveTimestamp.
- An important property which gets returned in WAQS is “PopReceipt” which must be used to either delete delete that message or update that message. Similarly in ASQS, property equivalent to “PopReceipt” in WAQS is “ReceiptHandle” which must be used to delete that particular message.
One thing that was not very clear to me as far as ASQS is concerned is the following comment in the documentation, and I quote:
”Due to the distributed nature of the queue, a weighted random set of machines is sampled on a ReceiveMessage call. That means only the messages on the sampled machines are returned. If the number of messages in the queue is small (less than 1000), it is likely you will get fewer messages than you requested per ReceiveMessage call. If the number of messages in the queue is extremely small, you might not receive any messages in a particular ReceiveMessage response; in which case you should repeat the request.”
If I am understanding it correctly, I might have to make a request multiple times in ASQS if my queue contains fewer messages. IMHO, I find this somewhat odd. Does this mean that if I have built a highly efficient system, where susbscribers are effectively and efficiently pulling messages off of a queue (read message count is lower), eventually over a period of time I may end up making multiple requests to get messages just because the number of messages in my queue are small.
Peek Messages
| WAQS | ASQS | |
| Peek Messages | Yes | No |
At times you just want to view the contents of your messages in a queue without hiding them. This functionality does just that. Essentially it is similar to Get Messages functionality but when you fetch messages using this functionality, the messages remain visible to all subscribers. Like Get Messages functionality you can request up to 32 messages from a queue in WAQS.
I mentioned above that this functionality is not available in ASQS. Well, that’s not entirely true. To be honest, when I first started looking into ASQS and could not find this functionality, I was kind of surprised. I posted a question on Twitter and got the following response:
Essentially this functionality was there initially but was removed in one of the later versions. Based on the comment here, even though this functionality is not available directly in the API you can achieve this by invoking ReceiveMessage functionality and pass “0” as value for VisibilityTimeout parameter.
Delete Message/DeleteMessage
| WAQS | ASQS | |
| Delete Message/DeleteMessage | Yes | Yes |
As the name suggests, you can use this functionality to delete a message from a queue. This feature is supported in both systems. In both systems, in order to delete a message you would first need to fetch the message using Get Messages (WAQS) and ReceiveMessage (ASQS) functionality. In WAQS, you would need both MessageId and PopReceipt parameters to delete a message. Both of these are returned by “Get Messages” functionality. In ASQS, you would need ReceiptHandle parameter which is returned by “ReceiveMessage” functionality.
DeleteMessageBatch
| WAQS | ASQS | |
| DeleteMessageBatch | No | Yes |
This functionality allows you to delete one or more messages from a queue in a single call to the service. This functionality is only available in ASQS. Again, to delete messages in a batch you would need ReceiptHandle of the messages which can be obtained by invoking ReceiveMessage functionality. Even though the name of the functionality contains “batch” it does not mean that it is in transaction scope where either all messages are deleted or none are deleted. It is quite possible that when deleting messages by invoking this functionality, some of the messages are not deleted.
IMHO, it would be a very valuable addition to WAQS.
Update Message
| WAQS | ASQS | |
| Update Message | Yes | No |
Update message functionality allows you to change message content as well as message visibility timeout. In order to update a message, you would first need to fetch the message using Get Messages functionality. As to why you would need this functionality, here is one scenario: Let’s say you get the message using Get Messages functionality and at that time you made the message invisible for 30 seconds assuming you will be done with the message by that time. However for some reason half way through you figured out that you will not be done with processing the message in 30 seconds and would need more time. You could then invoke this functionality to extend the message visibility for say additional 30 seconds to ensure that this message remains invisible to other subscribers.
I mentioned above that this functionality is not available in ASQS. Well that’s partly true. While in ASQS you cannot change a message’s content after it has been created however you could change its visibility timeout by invoking either ChangeMessageVisibility or ChangeMessageVisibilityBatch functionality which are described below.
Clear Messages
| WAQS | ASQS | |
| Clear Messages | Yes | No |
Clear Messages functionality essentially empties a queue by deleting all messages from a queue. This functionality could come in handy when you just want to remove messages from a queue without deleting the queue. Please note that deleting a queue is an asynchronous process so if you delete a queue, depending on the number of messages in a queue the operation may take a longer time and during that time you will not be able to create another queue with the same name. In those scenarios, Clear Messages operation is preferable.
This functionality is not available in ASQS. There are two alternatives though: One, delete the queue and recreate it. Given that it may take up to 60 seconds for a queue to get deleted, in certain scenarios this may not work out well. Other option would be to fetch messages from the queue using ReceiveMessage functionality and then delete them either by using DeleteMessage or DeleteMessageBatch functionality. With this approach, there may be issues with timing and transaction costs.
ChangeMessageVisibility
| WAQS | ASQS | |
| ChangeMessageVisibility | Yes | Yes |
This functionality allows you to change visibility timeout of a message in a queue. It offers part of the functionality offered by Update Message functionality in WAQS.
ChangeMessageVisibilityBatch
| WAQS | ASQS | |
| ChangeMessageVisibilityBatch | No | Yes |
This functionality allows you to change visibility timeout of up to 10 messages in a queue in a single request in ASQS. There is no equivalent function available in WAQS. In WAQS, if you wish to change the visibility timeout of multiple messages, you would need to call Update Message functionality multiple times which will have transaction cost implications.
AddPermission
| WAQS | ASQS | |
| AddPermission | No | Yes |
This functionality allows you to grant permissions on a queue to another principal which allows you to share your queue with other users (by default, as owner of a queue only you have permissions to perform actions on a queue). When you invoke this function, ASQS writes an SQS-generated policy (read system defined). If you wish to create custom policy on a queue, you would need to use SetQueueAttributes functionality.
This functionality is not available in WAQS. IMHO, it’s a very neat functionality and is a much desired feature in ASQS.
RemovePermission
| WAQS | ASQS | |
| RemovePermission | No | Yes |
This functionality allows you to revoke any permissions you as a queue owner granted in the queue policy.
This functionality is not available in WAQS. IMHO, it’s a very neat functionality and is a much desired feature in ASQS.
Summary
To summarize, both systems are quite comparable and offer similar feature sets. There are minor differences in the functionality and if we as developers can keep them in mind, it is quite possible to build a system which abstracts both services so that it can be interchanged if needed.
Also when it comes to pricing, both systems are quite competitive and offer nearly similar pricing.
A few comments (Disclaimers)
- It is not the intent of the blog post to prove that one service is superior over other. I just wanted to have a very objective comparison of these two services.
- I make my living using Windows Azure Platform (if you wish, you can call me Windows Azure fan boy
). This does not mean that I have negative things to say about Amazon Cloud Computing Platform. It’s just that I never had a chance to play with Amazon just yet. - Since I haven’t played with Amazon platform just yet, the information I have presented in this blog post about Amazon SQS is solely based on my understanding of this service based on the documentation. Very likely, I may be wrong about some of things I have written here. Please note that it is not done intentionally and you can blame my lack of knowledge for that. If you find such issues, please let me know and I will fix them ASAP.
