For a change, this would be a technical post which will not be about Windows Azure :). For the service we are building at Cynapta, we had a requirement where we needed to upload large files using a pure browser based interface. For that we used Query String authentication mechanism available in Amazon S3 which provided time-limited access to buckets in Amazon S3 (in other words Pre Signed URL). In this blog post, we will talk about how we accomplished this using C#.
How Large Files are Uploaded in Amazon S3
Before we talk about using Query String authentication in Amazon S3, let’s take a moment and talk about how large files are uploaded in Amazon S3 and then we will focus on the issue at hand.
In order to upload large files, Amazon S3 provides a mechanism called “Multi Part Upload”. Using this mechanism, essentially you chunk a large file into smaller pieces (called “Parts” in Amazon S3 terminology) and upload these chunks. Once all parts are uploaded, you tell Amazon S3 to join these files together and create the desired object.
To do a “Multi Part Upload”, one would go through the following steps:
Initiate Multipart Upload
This is where you basically tell Amazon S3 that you will be uploading a large file. When Amazon S3 receives this request, it sends back an “Upload Id” which you have to use in subsequent requests. To learn more about this process, please click here: http://docs.aws.amazon.com/AmazonS3/latest/API/mpUploadInitiate.html.
Upload Part
This is where you split the large file into parts and upload these parts. A few things to remember are:
- Each part must be at least 5 MB in size with the exception of last part.
- Each part is assigned a sequential part number (starting from 1) and there can be a maximum of ten thousand (10,000) parts. In other words maximum number of parts in which a file can be split is ten thousand.
To learn more about this process, please click here: http://docs.aws.amazon.com/AmazonS3/latest/API/mpUploadUploadPart.html.
Complete Multipart Upload
Once all parts are uploaded, using this step you basically tell Amazon S3 to join all the parts together to create the object. To learn more about this process, please click here: http://docs.aws.amazon.com/AmazonS3/latest/API/mpUploadComplete.html.
Challenge
For the rest of the service, we used Amazon SDK for .Net and more or less it worked really well for us. For normal uploads where we didn’t have to split the file in parts, the SDK worked really great as it provides a function to create Pre Signed URL. However one thing we noticed is that there was no support for generating Pre Signed URL when you wanted to do Multi Part Upload and unfortunately I was not able to find code in .Net (C#) to do so.
So we ended up consuming Amazon S3 REST API in C# code to create Pre Signed URL.
Code
Let’s get into the code right away. What I did was created a console application and wrote some helper functions which implemented REST API for Amazon S3. For the purpose of demonstration, let’s assume that we want to upload a file called “verylargefile.exe” in a bucket called “gaurav-test-bucket” which is hosted in “us-west-2” region. Based on this, my object URL will be “https://gaurav-test-bucket.s3-us-west-2.amazonaws.com/verylargefile.exe”. Since we are uploading using Pre Signed URL, we will assume that the URL will be valid for 12 hours from the time we started the process.
Creating Authorization Header
First thing that we need to do is write code to create authorization header. To learn more about how to create authorization header, please click here: http://docs.aws.amazon.com/AmazonS3/latest/dev/RESTAuthentication.html. If you go through the documentation, you will realize that in order to create authorization header, there are a few things you would need to do:
- Create “CanonicalizedResource” Element String
- Create “CanonicalizedAmzHeaders” Element String
- Create “StringToSign String
- Create Signature
Now we’ll see the code for each of these steps:
Create “CanonicalizedResource” Element String
To create CanonicalizedResource Element String, you would need the request URI. Here’s the code to create that:
private static string[] subResourcesToConsider = new string[] { "acl", "lifecycle", "location", "logging", "notification", "partNumber", "policy", "requestPayment", "torrent", "uploadId", "uploads", "versionId", "versioning", "versions", "website", };
private static string[] overrideResponseHeadersToConsider = new string[] { "response-content-type", "response-content-language", "response-expires", "response-cache-control", "response-content-disposition", "response-content-encoding" };
private static string GetCanonicalizedResourceString(Uri requestUri)
{
var host = requestUri.DnsSafeHost;
var hostElementsArray = host.Split('.');
var bucketName = "";
if (hostElementsArray.Length > 3)
{
StringBuilder sb = new StringBuilder();
for (int i = 0; i < hostElementsArray.Length - 3; i++)
{
sb.AppendFormat("{0}.", hostElementsArray[i]);
}
bucketName = sb.ToString();
if (bucketName.Length > 0)
{
if (bucketName.EndsWith("."))
{
bucketName = bucketName.Substring(0, bucketName.Length - 1);
}
bucketName = string.Format("/{0}", bucketName);
}
}
var subResourcesList = subResourcesToConsider.ToList();
var overrideResponseHeadersList = overrideResponseHeadersToConsider.ToList();
StringBuilder canonicalizedResourceStringBuilder = new StringBuilder();
canonicalizedResourceStringBuilder.Append(bucketName);
canonicalizedResourceStringBuilder.Append(requestUri.AbsolutePath);
NameValueCollection queryVariables = HttpUtility.ParseQueryString(requestUri.Query);
SortedDictionary<string, string> queryVariablesToConsider = new SortedDictionary<string, string>();
SortedDictionary<string, string> overrideResponseHeaders = new SortedDictionary<string, string>();
if (queryVariables != null && queryVariables.Count > 0)
{
var numQueryItems = queryVariables.Count;
for (int i = 0; i < numQueryItems; i++)
{
var key = queryVariables.GetKey(i);
var value = queryVariables[key];
if (subResourcesList.Contains(key))
{
if (queryVariablesToConsider.ContainsKey(key))
{
var val = queryVariablesToConsider[key];
queryVariablesToConsider[key] = string.Format("{0},{1}", value, val);
}
else
{
queryVariablesToConsider.Add(key, value);
}
}
if (overrideResponseHeadersList.Contains(key))
{
overrideResponseHeaders.Add(key, HttpUtility.UrlDecode(value));
}
}
}
if (queryVariablesToConsider.Count > 0 || overrideResponseHeaders.Count > 0)
{
StringBuilder queryStringInCanonicalizedResourceString = new StringBuilder();
queryStringInCanonicalizedResourceString.Append("?");
for (int i = 0; i < queryVariablesToConsider.Count; i++)
{
var key = queryVariablesToConsider.Keys.ElementAt(i);
var value = queryVariablesToConsider.Values.ElementAt(i);
if (!string.IsNullOrWhiteSpace(value))
{
queryStringInCanonicalizedResourceString.AppendFormat("{0}={1}&", key, value);
}
else
{
queryStringInCanonicalizedResourceString.AppendFormat("{0}&", key);
}
}
for (int i = 0; i < overrideResponseHeaders.Count; i++)
{
var key = overrideResponseHeaders.Keys.ElementAt(i);
var value = overrideResponseHeaders.Values.ElementAt(i);
queryStringInCanonicalizedResourceString.AppendFormat("{0}={1}&", key, value);
}
var str = queryStringInCanonicalizedResourceString.ToString();
if (str.EndsWith("&"))
{
str = str.Substring(0, str.Length - 1);
}
canonicalizedResourceStringBuilder.Append(str);
}
return canonicalizedResourceStringBuilder.ToString();
}
Create “CanonicalizedAmzHeaders” Element String
To create CanonicalizedAmzHeaders Element String, you would need all the headers which will be included in the request. Based on the documentation, only the headers starting with “x-amz-“ will be considered in this function though. Here’s the code to create that:
private static string GetCanonicalizedAmzHeadersString(NameValueCollection requestHeaders)
{
var canonicalizedAmzHeadersString = string.Empty;
if (requestHeaders != null && requestHeaders.Count > 0)
{
StringBuilder sb = new StringBuilder();
SortedDictionary<string, string> sortedRequestHeaders = new SortedDictionary<string, string>();
var requestHeadersCount = requestHeaders.Count;
for (int i = 0; i < requestHeadersCount; i++)
{
var key = requestHeaders.Keys.Get(i);
var value = requestHeaders[key].Trim();
key = key.ToLowerInvariant();
if (key.StartsWith("x-amz-", StringComparison.InvariantCultureIgnoreCase))
{
if (sortedRequestHeaders.ContainsKey(key))
{
var val = sortedRequestHeaders[key];
sortedRequestHeaders[key] = string.Format("{0},{1}", val, value);
}
else
{
sortedRequestHeaders.Add(key, value);
}
}
}
if (sortedRequestHeaders.Count > 0)
{
foreach (var item in sortedRequestHeaders)
{
sb.AppendFormat("{0}:{1}\n", item.Key, item.Value);
}
canonicalizedAmzHeadersString = sb.ToString();
}
}
return canonicalizedAmzHeadersString;
}
Create “StringToSign” String
There will be 2 methods that we need to implement for creating “StringToSign” string – one for regular authorization header and other one for authorization header for Pre Signed URL. The difference between the two is that in the former we have to pass the current date/time (in UTC) while in the latter we need to pass the number of seconds since Jan 1st 1970 during which the Pre Signed URL will be valid.
StringToSign for regular authorization header
Here’s the code to create StringToSign for regular authorization header:
private static string GetStringToSign(Uri requestUri, string httpVerb, string contentMD5, string contentType, DateTime date, NameValueCollection requestHeaders)
{
var canonicalizedResourceString = GetCanonicalizedResourceString(requestUri);
var canonicalizedAmzHeadersString = GetCanonicalizedAmzHeadersString(requestHeaders);
var dateInStringFormat = date.ToString("R");
if (requestHeaders != null && requestHeaders.AllKeys.Contains("x-amz-date"))
{
dateInStringFormat = string.Empty;
}
var stringToSign = string.Format("{0}\n{1}\n{2}\n{3}\n{4}{5}", httpVerb, contentMD5, contentType, dateInStringFormat, canonicalizedAmzHeadersString, canonicalizedResourceString);
return stringToSign;
}
StringToSign for Pre Signed URL authorization header
Here’s the code to create StringToSign for Pre Signed URL authorization header:
private static string GetStringToSign(Uri requestUri, string httpVerb, string contentMD5, string contentType, long secondsSince1stJan1970, NameValueCollection requestHeaders)
{
var canonicalizedResourceString = GetCanonicalizedResourceString(requestUri);
var canonicalizedAmzHeadersString = GetCanonicalizedAmzHeadersString(requestHeaders);
var stringToSign = string.Format("{0}\n{1}\n{2}\n{3}\n{4}{5}", httpVerb, contentMD5, contentType, secondsSince1stJan1970, canonicalizedAmzHeadersString, canonicalizedResourceString);
return stringToSign;
}
Create Signature
Last step in this process is to create signature. Here’s the code to do the same:
private static string CreateSignature(string secretKey, string stringToSign)
{
byte[] dataToSign = Encoding.UTF8.GetBytes(stringToSign);
using (HMACSHA1 hmacsha1 = new HMACSHA1(Encoding.UTF8.GetBytes(secretKey)))
{
return Convert.ToBase64String(hmacsha1.ComputeHash(dataToSign));
}
}
Once the signature is created, most of the heavy weight work is done :). Now we can focus on multipart upload.
Initiate Multipart Upload
As mentioned above, this process returns you an upload id. For this process, we would need to append “uploads=” as the query string to my object URL. Here’s the code to do so:
private static string InitiateMultipartUpload(string accessKey, string secretKey, Uri requestUri, DateTime requestDate, string contentType, NameValueCollection requestHeaders)
{
var uploadId = string.Empty;
var uploadIdRequestUrl = new Uri(string.Format("{0}?uploads=", requestUri.AbsoluteUri));
var uploadIdRequestUrlRequestHeaders = new NameValueCollection();
if (requestHeaders != null)
{
for (int i = 0; i < requestHeaders.Count; i++)
{
var key = requestHeaders.Keys[i];
var value = requestHeaders[key];
if (key.StartsWith("x-amz-", StringComparison.InvariantCultureIgnoreCase))
{
uploadIdRequestUrlRequestHeaders.Add(key, value);
}
}
}
var stringToSign = GetStringToSign(uploadIdRequestUrl, "POST", string.Empty, contentType, requestDate, requestHeaders);
var signatureForUploadId = CreateSignature(secretKey, stringToSign);
uploadIdRequestUrlRequestHeaders.Add("Authorization", string.Format("AWS {0}:{1}", accessKey, signatureForUploadId));
var request = (HttpWebRequest)WebRequest.Create(uploadIdRequestUrl);
request.Method = "POST";
request.ContentLength = 0;
request.Date = requestDate;
request.ContentType = contentType;
request.Headers.Add(uploadIdRequestUrlRequestHeaders);
using (var resp = (HttpWebResponse)request.GetResponse())
{
using (var s = new StreamReader(resp.GetResponseStream()))
{
var response = s.ReadToEnd();
XElement xe = XElement.Parse(response);
uploadId = xe.Element(XName.Get("UploadId", "http://s3.amazonaws.com/doc/2006-03-01/")).Value;
}
}
return uploadId;
}
A few things to consider here:
- If you want to set the content type and other properties of the object, this is the step you would do that. After this step, you won’t be able to set the content type during the upload process.
- Same thing goes for setting custom metadata attributes as well.
Upload Part
Once we get the upload id, next step would be to upload parts. For the purpose of our demonstration, we will split the file in 5 MB parts. Here’s the code to upload all parts.
private static Dictionary<int, string> UploadParts(string accessKey, string secretKey, Uri requestUri, string uploadId, string filePath, DateTime expiryDate)
{
Dictionary<int, string> partNumberETags = new Dictionary<int, string>();
DateTime Jan1st1970 = new DateTime(1970, 1, 1, 0, 0, 0, DateTimeKind.Utc);
TimeSpan ts = new TimeSpan(expiryDate.Ticks - Jan1st1970.Ticks);
var expiry = Convert.ToInt64(ts.TotalSeconds);
var fileContents = File.ReadAllBytes(filePath);
int fiveMB = 5 * 1024 * 1024;
int partNumber = 1;
var startPosition = 0;
var bytesToBeUploaded = fileContents.Length;
do
{
var bytesToUpload = Math.Min(fiveMB, bytesToBeUploaded);
var partUploadUrl = new Uri(string.Format("{0}?uploadId={1}&partNumber={2}", requestUri.AbsoluteUri, HttpUtility.UrlEncode(uploadId), partNumber));
var partUploadSignature = CreateSignature(secretKey, GetStringToSign(partUploadUrl, "PUT", string.Empty, string.Empty, expiry, null));
var partUploadPreSignedUrl = new Uri(string.Format("{0}?uploadId={1}&partNumber={2}&AWSAccessKeyId={3}&Signature={4}&Expires={5}", requestUri.AbsoluteUri,
HttpUtility.UrlEncode(uploadId), partNumber, accessKey, HttpUtility.UrlEncode(partUploadSignature), expiry));
var request = (HttpWebRequest)WebRequest.Create(partUploadPreSignedUrl);
request.Method = "PUT";
request.Timeout = 1000 * 600;
request.ContentLength = bytesToUpload;
using (var stream = request.GetRequestStream())
{
stream.Write(fileContents, startPosition, bytesToUpload);
}
using (var resp = (HttpWebResponse)request.GetResponse())
{
using (var s = new StreamReader(resp.GetResponseStream()))
{
partNumberETags.Add(partNumber, resp.Headers["ETag"]);
}
}
bytesToBeUploaded = bytesToBeUploaded - bytesToUpload;
startPosition = bytesToUpload;
partNumber = partNumber + 1;
}
while (bytesToBeUploaded > 0);
return partNumberETags;
}
Complete Multipart Upload
Once all the parts are uploaded, you would need to finish the process by performing complete multipart upload process. Here’s the code to do that.
private static void FinishMultipartUpload(string accessKey, string secretKey, Uri requestUri, string uploadId, Dictionary<int, string> partNumberETags, DateTime expiryDate)
{
DateTime Jan1st1970 = new DateTime(1970, 1, 1, 0, 0, 0, DateTimeKind.Utc);
TimeSpan ts = new TimeSpan(expiryDate.Ticks - Jan1st1970.Ticks);
var expiry = Convert.ToInt64(ts.TotalSeconds);
var finishOrCancelMultipartUploadUri = new Uri(string.Format("{0}?uploadId={1}", requestUri.AbsoluteUri, uploadId));
var signatureForFinishMultipartUpload = CreateSignature(secretKey, GetStringToSign(finishOrCancelMultipartUploadUri, "POST", string.Empty, "text/plain", expiry, null));
var finishMultipartUploadUrl = new Uri(string.Format("{0}?uploadId={1}&AWSAccessKeyId={2}&Signature={3}&Expires={4}", requestUri.AbsoluteUri, HttpUtility.UrlEncode(uploadId), accessKey, HttpUtility.UrlEncode(signatureForFinishMultipartUpload), expiry));
StringBuilder payload = new StringBuilder();
payload.Append("<?xml version=\"1.0\" encoding=\"utf-8\"?><CompleteMultipartUpload>");
foreach (var item in partNumberETags)
{
payload.AppendFormat("<Part><PartNumber>{0}</PartNumber><ETag>{1}</ETag></Part>", item.Key, item.Value);
}
payload.Append("</CompleteMultipartUpload>");
var requestPayload = Encoding.UTF8.GetBytes(payload.ToString());
var request = (HttpWebRequest)WebRequest.Create(finishMultipartUploadUrl);
request.Method = "POST";
request.ContentType = "text/plain";
request.ContentLength = requestPayload.Length;
using (var stream = request.GetRequestStream())
{
stream.Write(requestPayload, 0, requestPayload.Length);
}
using (var resp = (HttpWebResponse)request.GetResponse())
{
}
}
Complete Code
Here’s the complete code:
using System;
using System.Collections.Generic;
using System.Collections.Specialized;
using System.Globalization;
using System.IO;
using System.Linq;
using System.Net;
using System.Security.Cryptography;
using System.Text;
using System.Threading.Tasks;
using System.Web;
using System.Xml.Linq;
namespace AmazonS3RestWrapper
{
class Program
{
private static string[] subResourcesToConsider = new string[] { "acl", "lifecycle", "location", "logging", "notification", "partNumber", "policy", "requestPayment", "torrent", "uploadId", "uploads", "versionId", "versioning", "versions", "website", };
private static string[] overrideResponseHeadersToConsider = new string[] { "response-content-type", "response-content-language", "response-expires", "response-cache-control", "response-content-disposition", "response-content-encoding" };
static void Main(string[] args)
{
var accessKey = "access key";
var secretKey = "secret key";
var requestUri = new Uri("https://gaurav-test-bucket.s3-us-west-2.amazonaws.com/verylargefile.exe");
var filePath = @"D:\verylargefile.exe";
var expiryDate = DateTime.UtcNow.AddHours(12);
var uploadId = InitiateMultipartUpload(accessKey, secretKey, requestUri, DateTime.UtcNow, "application/x-msdownload", null);
var partNumberETags = UploadParts(accessKey, secretKey, requestUri, uploadId, filePath, expiryDate);
FinishMultipartUpload(accessKey, secretKey, requestUri, uploadId, partNumberETags, expiryDate);
Console.WriteLine("File uploaded successfully. Press any key to terminate the application.");
Console.ReadLine();
}
private static string GetStringToSign(Uri requestUri, string httpVerb, string contentMD5, string contentType, DateTime date, NameValueCollection requestHeaders)
{
var canonicalizedResourceString = GetCanonicalizedResourceString(requestUri);
var canonicalizedAmzHeadersString = GetCanonicalizedAmzHeadersString(requestHeaders);
var dateInStringFormat = date.ToString("R");
if (requestHeaders != null && requestHeaders.AllKeys.Contains("x-amz-date"))
{
dateInStringFormat = string.Empty;
}
var stringToSign = string.Format("{0}\n{1}\n{2}\n{3}\n{4}{5}", httpVerb, contentMD5, contentType, dateInStringFormat, canonicalizedAmzHeadersString, canonicalizedResourceString);
return stringToSign;
}
private static string GetStringToSign(Uri requestUri, string httpVerb, string contentMD5, string contentType, long secondsSince1stJan1970, NameValueCollection requestHeaders)
{
var canonicalizedResourceString = GetCanonicalizedResourceString(requestUri);
var canonicalizedAmzHeadersString = GetCanonicalizedAmzHeadersString(requestHeaders);
var stringToSign = string.Format("{0}\n{1}\n{2}\n{3}\n{4}{5}", httpVerb, contentMD5, contentType, secondsSince1stJan1970, canonicalizedAmzHeadersString, canonicalizedResourceString);
return stringToSign;
}
private static string GetCanonicalizedResourceString(Uri requestUri)
{
var host = requestUri.DnsSafeHost;
var hostElementsArray = host.Split('.');
var bucketName = "";
if (hostElementsArray.Length > 3)
{
StringBuilder sb = new StringBuilder();
for (int i = 0; i < hostElementsArray.Length - 3; i++)
{
sb.AppendFormat("{0}.", hostElementsArray[i]);
}
bucketName = sb.ToString();
if (bucketName.Length > 0)
{
if (bucketName.EndsWith("."))
{
bucketName = bucketName.Substring(0, bucketName.Length - 1);
}
bucketName = string.Format("/{0}", bucketName);
}
}
var subResourcesList = subResourcesToConsider.ToList();
var overrideResponseHeadersList = overrideResponseHeadersToConsider.ToList();
StringBuilder canonicalizedResourceStringBuilder = new StringBuilder();
canonicalizedResourceStringBuilder.Append(bucketName);
canonicalizedResourceStringBuilder.Append(requestUri.AbsolutePath);
NameValueCollection queryVariables = HttpUtility.ParseQueryString(requestUri.Query);
SortedDictionary<string, string> queryVariablesToConsider = new SortedDictionary<string, string>();
SortedDictionary<string, string> overrideResponseHeaders = new SortedDictionary<string, string>();
if (queryVariables != null && queryVariables.Count > 0)
{
var numQueryItems = queryVariables.Count;
for (int i = 0; i < numQueryItems; i++)
{
var key = queryVariables.GetKey(i);
var value = queryVariables[key];
if (subResourcesList.Contains(key))
{
if (queryVariablesToConsider.ContainsKey(key))
{
var val = queryVariablesToConsider[key];
queryVariablesToConsider[key] = string.Format("{0},{1}", value, val);
}
else
{
queryVariablesToConsider.Add(key, value);
}
}
if (overrideResponseHeadersList.Contains(key))
{
overrideResponseHeaders.Add(key, HttpUtility.UrlDecode(value));
}
}
}
if (queryVariablesToConsider.Count > 0 || overrideResponseHeaders.Count > 0)
{
StringBuilder queryStringInCanonicalizedResourceString = new StringBuilder();
queryStringInCanonicalizedResourceString.Append("?");
for (int i = 0; i < queryVariablesToConsider.Count; i++)
{
var key = queryVariablesToConsider.Keys.ElementAt(i);
var value = queryVariablesToConsider.Values.ElementAt(i);
if (!string.IsNullOrWhiteSpace(value))
{
queryStringInCanonicalizedResourceString.AppendFormat("{0}={1}&", key, value);
}
else
{
queryStringInCanonicalizedResourceString.AppendFormat("{0}&", key);
}
}
for (int i = 0; i < overrideResponseHeaders.Count; i++)
{
var key = overrideResponseHeaders.Keys.ElementAt(i);
var value = overrideResponseHeaders.Values.ElementAt(i);
queryStringInCanonicalizedResourceString.AppendFormat("{0}={1}&", key, value);
}
var str = queryStringInCanonicalizedResourceString.ToString();
if (str.EndsWith("&"))
{
str = str.Substring(0, str.Length - 1);
}
canonicalizedResourceStringBuilder.Append(str);
}
return canonicalizedResourceStringBuilder.ToString();
}
private static string GetCanonicalizedAmzHeadersString(NameValueCollection requestHeaders)
{
var canonicalizedAmzHeadersString = string.Empty;
if (requestHeaders != null && requestHeaders.Count > 0)
{
StringBuilder sb = new StringBuilder();
SortedDictionary<string, string> sortedRequestHeaders = new SortedDictionary<string, string>();
var requestHeadersCount = requestHeaders.Count;
for (int i = 0; i < requestHeadersCount; i++)
{
var key = requestHeaders.Keys.Get(i);
var value = requestHeaders[key].Trim();
key = key.ToLowerInvariant();
if (key.StartsWith("x-amz-", StringComparison.InvariantCultureIgnoreCase))
{
if (sortedRequestHeaders.ContainsKey(key))
{
var val = sortedRequestHeaders[key];
sortedRequestHeaders[key] = string.Format("{0},{1}", val, value);
}
else
{
sortedRequestHeaders.Add(key, value);
}
}
}
if (sortedRequestHeaders.Count > 0)
{
foreach (var item in sortedRequestHeaders)
{
sb.AppendFormat("{0}:{1}\n", item.Key, item.Value);
}
canonicalizedAmzHeadersString = sb.ToString();
}
}
return canonicalizedAmzHeadersString;
}
private static string CreateSignature(string secretKey, string stringToSign)
{
byte[] dataToSign = Encoding.UTF8.GetBytes(stringToSign);
using (HMACSHA1 hmacsha1 = new HMACSHA1(Encoding.UTF8.GetBytes(secretKey)))
{
return Convert.ToBase64String(hmacsha1.ComputeHash(dataToSign));
}
}
private static string InitiateMultipartUpload(string accessKey, string secretKey, Uri requestUri, DateTime requestDate, string contentType, NameValueCollection requestHeaders)
{
var uploadId = string.Empty;
var uploadIdRequestUrl = new Uri(string.Format("{0}?uploads=", requestUri.AbsoluteUri));
var uploadIdRequestUrlRequestHeaders = new NameValueCollection();
if (requestHeaders != null)
{
for (int i = 0; i < requestHeaders.Count; i++)
{
var key = requestHeaders.Keys[i];
var value = requestHeaders[key];
if (key.StartsWith("x-amz-", StringComparison.InvariantCultureIgnoreCase))
{
uploadIdRequestUrlRequestHeaders.Add(key, value);
}
}
}
var stringToSign = GetStringToSign(uploadIdRequestUrl, "POST", string.Empty, contentType, requestDate, requestHeaders);
var signatureForUploadId = CreateSignature(secretKey, stringToSign);
uploadIdRequestUrlRequestHeaders.Add("Authorization", string.Format("AWS {0}:{1}", accessKey, signatureForUploadId));
var request = (HttpWebRequest)WebRequest.Create(uploadIdRequestUrl);
request.Method = "POST";
request.ContentLength = 0;
request.Date = requestDate;
request.ContentType = contentType;
request.Headers.Add(uploadIdRequestUrlRequestHeaders);
using (var resp = (HttpWebResponse)request.GetResponse())
{
using (var s = new StreamReader(resp.GetResponseStream()))
{
var response = s.ReadToEnd();
XElement xe = XElement.Parse(response);
uploadId = xe.Element(XName.Get("UploadId", "http://s3.amazonaws.com/doc/2006-03-01/")).Value;
}
}
return uploadId;
}
private static Dictionary<int, string> UploadParts(string accessKey, string secretKey, Uri requestUri, string uploadId, string filePath, DateTime expiryDate)
{
Dictionary<int, string> partNumberETags = new Dictionary<int, string>();
DateTime Jan1st1970 = new DateTime(1970, 1, 1, 0, 0, 0, DateTimeKind.Utc);
TimeSpan ts = new TimeSpan(expiryDate.Ticks - Jan1st1970.Ticks);
var expiry = Convert.ToInt64(ts.TotalSeconds);
var fileContents = File.ReadAllBytes(filePath);
int fiveMB = 5 * 1024 * 1024;
int partNumber = 1;
var startPosition = 0;
var bytesToBeUploaded = fileContents.Length;
do
{
var bytesToUpload = Math.Min(fiveMB, bytesToBeUploaded);
var partUploadUrl = new Uri(string.Format("{0}?uploadId={1}&partNumber={2}", requestUri.AbsoluteUri, HttpUtility.UrlEncode(uploadId), partNumber));
var partUploadSignature = CreateSignature(secretKey, GetStringToSign(partUploadUrl, "PUT", string.Empty, string.Empty, expiry, null));
var partUploadPreSignedUrl = new Uri(string.Format("{0}?uploadId={1}&partNumber={2}&AWSAccessKeyId={3}&Signature={4}&Expires={5}", requestUri.AbsoluteUri,
HttpUtility.UrlEncode(uploadId), partNumber, accessKey, HttpUtility.UrlEncode(partUploadSignature), expiry));
var request = (HttpWebRequest)WebRequest.Create(partUploadPreSignedUrl);
request.Method = "PUT";
request.Timeout = 1000 * 600;
request.ContentLength = bytesToUpload;
using (var stream = request.GetRequestStream())
{
stream.Write(fileContents, startPosition, bytesToUpload);
}
using (var resp = (HttpWebResponse)request.GetResponse())
{
using (var s = new StreamReader(resp.GetResponseStream()))
{
partNumberETags.Add(partNumber, resp.Headers["ETag"]);
}
}
bytesToBeUploaded = bytesToBeUploaded - bytesToUpload;
startPosition = bytesToUpload;
partNumber = partNumber + 1;
}
while (bytesToBeUploaded > 0);
return partNumberETags;
}
private static void FinishMultipartUpload(string accessKey, string secretKey, Uri requestUri, string uploadId, Dictionary<int, string> partNumberETags, DateTime expiryDate)
{
DateTime Jan1st1970 = new DateTime(1970, 1, 1, 0, 0, 0, DateTimeKind.Utc);
TimeSpan ts = new TimeSpan(expiryDate.Ticks - Jan1st1970.Ticks);
var expiry = Convert.ToInt64(ts.TotalSeconds);
var finishOrCancelMultipartUploadUri = new Uri(string.Format("{0}?uploadId={1}", requestUri.AbsoluteUri, uploadId));
var signatureForFinishMultipartUpload = CreateSignature(secretKey, GetStringToSign(finishOrCancelMultipartUploadUri, "POST", string.Empty, "text/plain", expiry, null));
var finishMultipartUploadUrl = new Uri(string.Format("{0}?uploadId={1}&AWSAccessKeyId={2}&Signature={3}&Expires={4}", requestUri.AbsoluteUri, HttpUtility.UrlEncode(uploadId), accessKey, HttpUtility.UrlEncode(signatureForFinishMultipartUpload), expiry));
StringBuilder payload = new StringBuilder();
payload.Append("<?xml version=\"1.0\" encoding=\"utf-8\"?><CompleteMultipartUpload>");
foreach (var item in partNumberETags)
{
payload.AppendFormat("<Part><PartNumber>{0}</PartNumber><ETag>{1}</ETag></Part>", item.Key, item.Value);
}
payload.Append("</CompleteMultipartUpload>");
var requestPayload = Encoding.UTF8.GetBytes(payload.ToString());
var request = (HttpWebRequest)WebRequest.Create(finishMultipartUploadUrl);
request.Method = "POST";
request.ContentType = "text/plain";
request.ContentLength = requestPayload.Length;
using (var stream = request.GetRequestStream())
{
stream.Write(requestPayload, 0, requestPayload.Length);
}
using (var resp = (HttpWebResponse)request.GetResponse())
{
}
}
}
}
Some Thoughts
Having worked with both Windows Azure (extensively I might add) and Amazon S3 (I’m just starting with it), I find uploading large files rather easy in Windows Azure. Here’re my reasons:
- Windows Azure lets you split the file in any chunk size whereas with Amazon S3, the part size must be a minimum of 5 MB. IMHO, having this large part size restriction could lead to more timeout exceptions. Considering the Internet speed here in India, it was quite painful to wait for 5 MB part to upload (Yeah, it’s that bad :)).
- With Windows Azure, you just create an upload Shared Access Signature on the blob container and you can upload any file there. However with Amazon S3, creating a Pre Signed URL on bucket won’t work. Your Pre Signed URL must include object key. Things get complicated if you’re uploading the file in parts. In that case when you create Pre Signed URL, you need to pass both Upload Id and Part Number in the signature creation process.
Summary
That’s it for this post. I hope you have found it useful. As I’m still learning Amazon S3, it is highly likely that I may have made some mistakes. If that’s the case, please let me know and I’ll fix them ASAP.
Happy Coding!!!
