Building a distributed web service using Amazon Web Services

A few months ago my employer asked me if it would be possible to create a web service for encoding videos. I had been playing around with Amazon’s web services for a while, and it seemed like the perfect foundation for building this.

I decided to build the backend in Python and use ffmpeg for encoding movies. I looked into building the web service frontend in Python as well, but the SOAP libraries I could find for Python did not seem very mature or maintained. Instead I started thinking about building it in ASP.NET (I had previous experiences from building web services in ASP.NET). After some research and testing with Apache and Mono (I wanted to use Linux VMs only) I decided to develop the frontend in ASP.NET but host it on Apache.

To make the service scalable I decided to break it down into several parts and use message passing between the different parts. The parts I broke it down into are:

• Web service frontend – what the user calls to encode a movie. Implemented in ASP.NET, hosted on Apache/Mono on Linux.
• Encode Worker – A python process managing the encoding of videos.
• Encode Master – Manages number of running Encode Workers. Implemented in Python.

When a movie gets uploaded to the web service frontend it gets placed into the encode queue. The encode workers periodically checks if there is anything in the queue, and if it is encodes it. The Encode Master manages the number of running Encode Workers (based on the current length of the encode queue). If the queue growes to long, we just fire up a few new VMs running the worker.

This is a schematic view of how the service has been implemented and how the different components are related to the Amazon services:

Right now we are implementing the solution for our first customer, pretty existing I must say! In an upcoming post I will discuss the different libraries for Python and ASP.NET I used for communicating with the Amazon Web Services.