In WSGI an application is just a callable object (remember that in Python functions are objects too!) that takes two parameters, environ and start_response. A very simple application could look like this:

Not very exiting but it shows us the very basics of WSGI. The function start_response is a function that, as the name implies, starts sending out the response. This is where you give status and headers. Lastly the application returns an iterator with the body response (usually a list of strings or a list containing one string that is the entire body). As you can see WSGI lets code pass around web request in a fairly formal way.

To run this application you can either install Apache and configure it with modwsgi, this might however be slightly overkill just in order to test the application. Instead I recommend installing Python Paste (pythonpaste.com), which is kind of a framework for web frameworks. It includes a lot of functionality that can be reused, but more importantly right now, it includes a simple web server that can serve WSGI applications. In Ubuntu 9.04 (the operating system I am currently running) Paste can be installed using apt:

To run the application, add the folowing lines to the end of your source file:

Now it should be possible to run the application:

Your web application is now up running! In the next part i will introduce you to a slightly more exiting application.

I decided to build the backend in Python and use ffmpeg for encoding movies. I looked into building the web service frontend in Python as well, but the SOAP libraries I could find for Python did not seem very mature or maintained. Instead I started thinking about building it in ASP.NET (I had previous experiences from building web services in ASP.NET). After some research and testing with Apache and Mono (I wanted to use Linux VMs only) I decided to develop the frontend in ASP.NET but host it on Apache.

To make the service scalable I decided to break it down into several parts and use message passing between the different parts. The parts I broke it down into are:

• Web service frontend – what the user calls to encode a movie. Implemented in ASP.NET, hosted on Apache/Mono on Linux.
• Encode Worker – A python process managing the encoding of videos.
• Encode Master – Manages number of running Encode Workers. Implemented in Python.

When a movie gets uploaded to the web service frontend it gets placed into the encode queue. The encode workers periodically checks if there is anything in the queue, and if it is encodes it. The Encode Master manages the number of running Encode Workers (based on the current length of the encode queue). If the queue growes to long, we just fire up a few new VMs running the worker.

This is a schematic view of how the service has been implemented and how the different components are related to the Amazon services:

Right now we are implementing the solution for our first customer, pretty existing I must say! In an upcoming post I will discuss the different libraries for Python and ASP.NET I used for communicating with the Amazon Web Services.

A solution to the problem that seemed a bit more challenging than using CURL and regular expressions, and which would be able to handle AJAX, was to program an existing web browser to visit the page and fetch the data from the web browser itself. I had programmed for KHTML before so i looked into if this was possible and found that Paul Giannaros had solved most of my problems. He had created PyKHTML:

PyKHTML is a Python module for writing website scrapers/spiders. Whereas traditional methods focus on writing the code to parse HTML/forms themselves, PyKHTML uses the excellent KHTML engine to do all the trudge work. It therefore handles web pages very well (even the severely crufty ones) and is pretty darn fast (implemented in C++). As a bonus, the module handles JavaScript and cookies transparently. Hurrah!

As an example for this post I decided to parse a digg article to find out who digged it. To understand the could you should know that KHTML uses a event driven programming model. This is my test program:

As you can see, the first thing we do is to create a PyHTML browser and loads the url.   load() takes two arguments: the url to be loaded and a function pointer to the function that should be executed when KHTML has loaded the url. To be able to provide this function a argument, I construct a lambda function.

So when the page has been loaded, processPage() is called. First we check if the page has completed loading, otherwise we wait some. When the page has completed loading, it is time to access the KHTML DOM data. PyHTML provides us quite a few nifty functions to access the DOM of KHTML, such as:

By accessing these functions we easily get access to the data we are interested in. To go to the next page, we get the the element with of the ‘nextprev’ class, and simply ‘clicks’ it by calling nextprev.click(). Then we do a recursive call to proccessPage() and processes the next page. When the program has terminated it will have listed all people who digged the article.