Gaurav Mantri's Personal Blog.

Uploading Large File By Splitting Into Blocks In Windows Azure Blob Storage Using Windows Azure SDK For PHP

In this blog post, we will see how you can upload a large blob in blob storage using Windows Azure SDK for PHP. I must state that I don’t know anything about PHP and did this exercise in order to help somebody out on StackOverflow. What helped me in the process is excellent documentation on PHP’s website and my knowledge on how Windows Azure Blob Storage Service REST API works.

What I realized during this process is that it’s fun to get out of one’s comfort zone (.Net for me) once in a while Smile. It’s extremely rewarding when you accomplish something.

Since I’m infamous for writing really long blog posts Smile, if you’re interested in seeing the final code, either scroll down below to the bottom of this post or head on to StackOverflow. Otherwise please read on Smile.

Since we’re uploading really large files, the procedure would be to split the file in chunks (blocks), upload these chunks and then commit those chunks.

Getting Started

I’m assuming you have installed Windows Azure SDK for PHP. If not you can download it from here: http://www.windowsazure.com/en-us/downloads/?sdk=php. This SDK depends on some external packages. For this blog post, only thing we need is to install Http_Request2 PEAR package which you can download from here: http://pear.php.net/package/HTTP_Request2.

Add Proper Classes

We just have to ensure that we have referenced all the classes we need in our code

<?php 
require_once 'WindowsAzure/WindowsAzure.php';
use WindowsAzure\Common\ServicesBuilder;
use WindowsAzure\Common\ServiceException;
use WindowsAzure\Blob\Models\Block;
use WindowsAzure\Blob\Models\BlobBlockType;
?>

Get Azure Things in place

This would mean creating an instance of “BlobRestProxy” class in our code. For the purpose of this blog, I’m uploading a file in storage emulator.

	$connectionString = "UseDevelopmentStorage=true";
	$instance = ServicesBuilder::getInstance();
	$blobRestProxy = $instance -> createBlobService($connectionString);
	$containerName = "[mycontainer]";
	$blobName = "[myblobname]";

Here’re the operations we would need to do:

Read file in chunks

To read file in chunks, first we’ll define the chunk size

define('CHUNK_SIZE', 1024*1024);//Block Size = 1 MB

Then we’ll get the file handler by specifying the file name and opening the file

$handler = fopen("[full file path]", "r");

and now we’ll read the file in chunks

	while (!feof($handler))
	{
		$data = fread($handler, CHUNK_SIZE);
	}
	fclose($handler); 

Prepare blocks

Before this, there are a few things I want to mention about blocks:

  • A file can be split into fifty thousand (50000) blocks.
  • Each block must be assigned a unique id (block id).
  • All block ids must have the same length. I would encourage you to read my previous blog for more details regarding this.
  • When sending to Windows Azure, each block id must be Base64 encoded.

Based on this, what we’re going to do is assign each block an incrementing integer value and to keep block id length the same, we’ll pad it with zeros so that the length of the block id is 6 characters.

		$counter = 1;
		$blockId = str_pad($counter, 6, "0", STR_PAD_LEFT);

Then we’ll create an instance of “Block” class and add that block id there with type as “Uncommitted”.

		$block = new Block();
		$block -> setBlockId(base64_encode($blockId));
		$block -> setType("Uncommitted");

Then we add this block to an array. This array will be used in the final step for committing the blocks (chunks).

		$blockIds = array();
		array_push($blockIds, $block);

Upload Blocks

Now that we have the chunk content ready, we just need to upload it. We will make use of “createBlobBlock” function in “BlobRestProxy” class to upload the block.

		$blobRestProxy -> createBlobBlock($containerName, $blobName, base64_encode($blockId), $data);

We would need to do this for each and every block we wish to upload.

Committing Blocks

This is the last step. Once all the blocks have been uploaded, we need to tell Windows Azure Blob Storage to create a blob by adding all blocks we’ve uploaded so far. We will make use of “commitBlobBlocks” function again in “BlobRestProxy” class to commit the block.

	$blobRestProxy -> commitBlobBlocks($containerName, $blobName, $blockIds);

That’s it! You should be able to see the blob in your blob storage. It’s that easy Smile.

Complete Code

Here’s the complete code:

<?php 
require_once 'WindowsAzure/WindowsAzure.php';
use WindowsAzure\Common\ServicesBuilder;
use WindowsAzure\Common\ServiceException;
use WindowsAzure\Blob\Models\Block;
use WindowsAzure\Blob\Models\BlobBlockType;
define('CHUNK_SIZE', 1024*1024);//Block Size = 1 MB
try {
	
	$connectionString = "UseDevelopmentStorage=true";
	$instance = ServicesBuilder::getInstance();
	$blobRestProxy = $instance -> createBlobService($connectionString);
	$containerName = "[mycontainer]";
	$blobName = "[myblobname]";
	$handler = fopen("[full file path]", "r");
	$counter = 1;
	$blockIds = array();
	while (!feof($handler))
	{
		$blockId = str_pad($counter, 6, "0", STR_PAD_LEFT);
		$block = new Block();
		$block -> setBlockId(base64_encode($blockId));
		$block -> setType("Uncommitted");
		array_push($blockIds, $block);
		$data = fread($handler, CHUNK_SIZE);
		echo " \n ";
		echo " -----------------------------------------";
		echo " \n ";
		echo "Read " . strlen($data) . " of data from file";
		echo " \n ";
		echo " -----------------------------------------";
		echo " \n ";
		echo "Uploading block #: " . $blockId . " into blob storage. Please wait.";
		echo " \n ";
		echo " -----------------------------------------";
		echo " \n ";
		$blobRestProxy -> createBlobBlock($containerName, $blobName, base64_encode($blockId), $data);
		echo "Uploaded block: " . $blockId . " into blob storage.";
		echo " \n ";
		echo " -----------------------------------------";
		echo " \n ";
		$counter = $counter + 1;
	}
	fclose($handler); 
	echo "Now committing block list. Please wait.";
	echo " \n ";
	echo " -----------------------------------------";
	echo " \n ";
	$blobRestProxy -> commitBlobBlocks($containerName, $blobName, $blockIds);
	echo "Blob created successfully.";
}
catch(Exception $e){
    // Handle exception based on error codes and messages.
    // Error codes and messages are here: 
    // http://msdn.microsoft.com/en-us/library/windowsazure/dd179439.aspx
    $code = $e->getCode();
    $error_message = $e->getMessage();
    echo $code.": ".$error_message."<br />";
}
?>

Summary

As you saw, how insanely easy it is to upload a large file in blob storage using PHP SDK. I didn’t (and still don’t) know anything about PHP but I was able to put this code together in a matter of hours. I think if you’re a PHP developer, you should be able to do so in minutes. I hope you’ve found this information useful. As always, if you find any issues with the post please let me know and I’ll fix it ASAP.

Happy Coding!!!

Comments

  1. Hi,

    Do you have same blog for .Net?

    Regards,
    Soumen Banerjee

  2. Jesper Larsen says:

    Very cool, thanks. I tried to figure this out for a while, but didn’t succeed, so it’s great with a sample of how to deal with this.

    Unfortunately this doesn’t seem to work. I’ve tried with a few large files, and they all hang at chunk 6. After a wait i get a “10: Malformed response:” error. This is running on OSX by the way.

  3. Jesper Larsen says:

    I should add another observation. It seems to be 6MB being a limit here. If i make smaller chunks, the code will upload more proportionally. Increasing the PHP memory limit does not seem to make a difference.

  4. Thank you this was very helpful.
    I did encounter on small problem, if the file size is an exact multiple of the chunk size, the last call to read returns zero bytes which then causes the upload to fail. It was easy enough to solve by moving the read to the top of the loop and check for zero bytes.

Speak Your Mind

*