source code bean

I recently needed to setup a new ASP.NET development environment and deiced to give Microsofts Web Platform Installer 2.0 a chance. I had actually never heard about this product before, but what it does is installing .NET, the needed development libraries, and the express versions of Visual Studio and SQL Server. It required three reboots, but in total it saved me a lot of time!

For the last month I have been playing around with WSGI in Python. WSGI is an interface between the web server and the web application, it is meant to simplify writing your own web framework in Python. My intention is not to write an fully fledged web framework, but rather just play around with some ideas I have and try it out.

In WSGI an application is just a callable object (remember that in Python functions are objects too!) that takes two parameters, environ and start_response. A very simple application could look like this:

Not very exiting but it shows us the very basics of WSGI. The function start_response is a function that, as the name implies, starts sending out the response. This is where you give status and headers. Lastly the application returns an iterator with the body response (usually a list of strings or a list containing one string that is the entire body). As you can see WSGI lets code pass around web request in a fairly formal way.

To run this application you can either install Apache and configure it with modwsgi, this might however be slightly overkill just in order to test the application. Instead I recommend installing Python Paste (pythonpaste.com), which is kind of a framework for web frameworks. It includes a lot of functionality that can be reused, but more importantly right now, it includes a simple web server that can serve WSGI applications. In Ubuntu 9.04 (the operating system I am currently running) Paste can be installed using apt:

To run the application, add the folowing lines to the end of your source file:

Now it should be possible to run the application:

Your web application is now up running! In the next part i will introduce you to a slightly more exiting application.

I was having a chat the other day with Danish Peter (yes, another one) about URL rewrites in ASP.NET. In a previous post I wrote about how to use friendly URLs in the Zend framework, in this post, which is based on our discussion, I will discuss how to do this in ASP.NET.

Friendly URLs are achieved by a technique called URL-rewrite. A friendly URL is a URL that is easy to read and understand. Let me give you an example (from a site my girlfriend loves) of what is NOT a friendly URL:

http://www.bebe.com/bebe-Jersey-Knit-Apron-Dress/dp/B001UX1NBQ?ie=UTF8&asinSearchPageIndex=13&pf_rd_r=1C6TS27BZZ1CAD6WY81S&navAsinList=B001UNM3XS%2CB001OPYKKQ%2CB001UNNQ52%2CB001RPD88I%
2CB001R4NHHQ%2CB0026BM39W%2CB001QOU6TY%2CB001UX1NNY%2CB001UAA4WI
%2CB0024QP6KM%2CB0026RRN4G%2CB0027DCI6W%2CB001UX3QL6%2CB001UX1NBQ
%2CB001VROPT8%2CB00265NBKS%2CB001RPM80C%2CB001UBWNTO%2CB001RPIC1G
%2CB001PA2NMW&node=675941011&pf_rd_s=search-results&field_browse=675941011&searchSize=20&navAsinListIndex=0&pf_rd_t=101&field_availability=0&id=bebe%20Jersey%20Knit%20Apron%20Dress&searchBinNameList=null&store=core&pf_rd_p=476815091&ref=search_results_14&searchNodeID=675941011&pf_rd_i=675941011&field_launch-date=-1y&searchRank=-custom-rank&searchPage=1&pf_rd_m=A2FMOXN01TSNYY

This might be an extreme example, but there are plenty of sites which are as bad. If the above URL was transformed into a friendly URL it might look something like:

http://www.bebe.com/Apparel/Dresses/Jersey-Knit-Apron-Dress.html

The second address is easier to read. URL rewrite techniques have been around for a long time and the first time I used it was back in the early Apache 1.3 days. Today good sites use friendly URLs and if you have not adopted it yet, you need to start using it.

There are several benefits from using a friendly url scheme, the most prominent are:

1. 1. The clear structure makes it easier for humans to read and understand the addres
2. 2. It is easier for search engines to understand what the address is and crawler the site. URL structure that goes more than three directories deep is not always read by a spider.
3. 3. If search engines understand the URL, it is also easier for them to help people finding what they want, giving better search results. Hence using keywords and proper titles makes life easier.

What options are there for ASP.NET?
There is no built in support for having friendly URLs in ASP.NET, but you could spend your time writing your own URL-rewrite module. Writing your own module is basically spending time reinventing the wheel. If you have the time, go ahead, but if not there are already several good ones out there (free and open source). One that I have found very useful is the URLRewriter.NET (http://urlrewriter.net/). URLRewriter.NET provides similar rewrite capabilities as the mod_rewrite does for Apache.

Using URLRewriter.NET
It is possible to use the URLRewriter.NET module without making any changes to the IIS configuration, this will however limit the functionallity the URLRewriter.NET rewriting capabilities. To use its full power, IIS should be configured to map all requests to the ASP.NET runtime (how to do this is covered in the manual). We will only be touching the surface of URLRewriter.NET so we will not have to do any modifications to the IIS configuration.

To add URLRewriter.NET to your project start out by opening Visual Studio or Visual Web Developer. Open your project and then right click the project in the solution explorer and click on Add Reference, go browse and find the folder you downloaded and unzipped. You will find the binary (.dll) file in “Loation”\urlrewriternet20rc1b6\UrlRewriterV2\bin\Release.

Before we can use the module we need to make some changes to the web.config. Add the following in the configSections:

This will allow the URL writer to read the rewrite rules we will define later. Now we need to add this module to the httpModules section:

By adding the rewriter as a httpModule we allow it to intercept web requests and route the requests to the actual aspx file serving the page. Now we need to define the rewrite rules:

The rewrite rules are defined using regular expressions. Regular expressions are very powerful and is something every programmer should know. The syntax might be hard to remember but there are several great tools to help build advanced regular expressions, for example the RegexDesigner.NET (http://www.sellsbrothers.com/tools/#regexd) that Chris Sells has created.

The above rewrite rules are simple. The first rewrite rule will rewrite URLs like

sourcecodebean.com/2009/04/01/1.shtml

to

sourcecodebean.com/Default.aspx?Year=2009&Month=04&Date=01&Id=1

And the second rewrite rule will match requests starting with ~/Article/ so it will rewrite URLs like

sourcecodebean.com/Article/1.shtml

to

sourcecodebean.com/Articles.aspx?ID=1

Finishing up
By now you hopefully have a functioning site that uses friendly URLs. One of the common problems you might run into is broken links. To avoid this you should make sure that you reference all your pages from the root level (~) and that all your links has the runat=server tag.

Thanks for your input Peter! You can check out his Danish Peter blog over at Magic Mouse.

For the last three months I have been working with one of Sweden’s largest companies. When I arrived the first day I noticed that it was not possible to log in to the intranet using Firefox. The explanation I received:

“our sites are built for Internet Explorer, you have to use it”

In my head a warning bell rang. I fired up Internet Explorer to find that the whole site was a horrible mix of broken HTML, badly written Javascript, lot of frames and almost no use of CSS. It surly could not work in anything else other than Internet Explorer 6 and 7.

My job was not the development of the intranet (which I am glad for), so I just told the customer what I thought and that they should do a better job of following web standards so the users could use the browser of their choice. The kind of stupid response I got was something in style with:

“we concentrate on Microsoft technologies, so the only requirement we have is that it should work in Internet Explorer”

The fun started during my last week at the company, when Microsoft started to push out Internet Explorer 8 to all Windows Update users. IE8 is actually very strict when it comes to following web standards. Lo and behold, none of the users who had their systems upgraded could use the intranet. IE8, now strict on standards, vomited back the bad intranet and the company basically came to standstill. Chaos!

I observed the chaos (smiling) for an hour before I decided to save the day by telling them about “EmulateIE7″. By default, IE8 uses the new IE8 rendering engine. The big difference from before is that if a site is broken IE8 will NOT default to an older rendering engine. This is different from how older IE browsers behaved and is clearly a step in the right direction for Microsoft to support web standards. However they have not scraped backwards compatibility entirely. In order to allow for broken sites to be viewed in IE8, it is possible for the developer to add the following meta-tag to the page:

This tag will tell IE8 to use the IE7 rendering engine.

And if you are stubborn enough to believe that building websites for browsers is acceptable, there are reasons why we need web standards:

1. 1. Accessibility is the law: FCC regulations in the United States require that web sites are accessible to people with disabilities. If a website cannot be viewed by someone with an impairment, it is not only illegal but highly unfair to visitors. With proper web standards, these visitors can change font and modify a website layout to make it easier to read.
2. 2. Better search engine rankings: While people may be gung ho for “SEO” and those techniques, some of the most basic and most important ways of reading a website can improve the reading by a search engine spider. Combined with semantic markup, the separation of content and presentation generally improves search engine rankings.
3. 3. Easier development and maintenance: Using semantic and structured HTML makes it easier and quicker to understand code created by somebody else. Since every programmer leaves a footprint in their style of code, those that write web standards will make it easy for others to understand code. It will also help avoid the hackneyed job that many sites exist in today.
4. 4. CSS = Faster downloading and rendering of pages: Using CSS for presentation style leads to leaner HTML documents. It also makes it easier to globally change attributes. Leaner files means quicker downloads for visitors.
5. 5. Semantic HTML is friendly to adaptation: Semantically marked up documents can beadapted to print and alternative browsing devices (PDAs, iPhones, netbooks) by linking to a different CSS file.

After the incident was resolved we had a long discussing about web standards, and the company now understands that you can not design a site for a certain browser, it should be designed to follow accepted web standards.

I first learned about functional programming a few years ago when I took a class in Haskell programming at my university, Chalmers. While Haskell might not be for everyday use, many of the concepts are very compelling, such as higher order functions, pattern matching, lazy evaluation, list comprehensions, lambda functions and map/fold/filter. Over the years many mainstream programming languages have adopted a subset of those features. With the release of C# 3.0 Microsoft has brought some of this functionality to C#.

Lambda functions
A lambda function is simply an anonymous function that can contain expressions and statements. Lambda functions can be used to create delegates or expression trees. Lambda functions use the lambda operator => (read as “goes to”). On the left side the input parameters is defined and the right side holds the expressions or statements. A simple example would be:

But wait, where is the type declaration? The types does not have to be defined, the compiler will declare the types by using type interference at compile time, so everything is still strongly typed.

The framework defines a number of parameterized delegate types (the first type is the return type):

So we could define our function as follows:

Map/fold/filter
In functional programming it is very common to work with lists. Three very common computations are:

1. Applying a function to every element of a list (map)

2. Computing a value based on all elements in the list. For example the total sum of a list containing numeric values (fold)

3. Selecting just a subset of the elements (filter)

In C# the equivalents are Select, Aggregate and Where. Here is a simple example of how they can be used:

Using map/fold/filter instead of using regular loops makes it easier to understand and reason your code.

You would think that ASP.NET would have build in support for something so simple as clearing all input fields on a page, but no. However, this easy to add. Add the following to your Page_Load (btnClear should be the name of your clear button):


btnClear.Attributes.Add("onClick", "document.forms[0].reset();return false;");


The workaround I tested a few weeks ago did not really solve my problem – actually it made it worse. The autorestart caused mono to hang and not restart at all, so my site stopped responding every 6th hour. I quickly had to disable this. Still I had the memory consumption problems. From various sources I was told that the memory issues would be fixed in the recently released mono 2.4. Also I found a bug report regarding AutoRestart, which also should have been fixed in 2.4. So I decided to give Mono 2.4 a try.
The problem was that there were no packages for Ubuntu 8.10, so I had to download the sources and build my own deb packages. I found this great blog post that describes the process of building and installing Mono 2.4.

So far it seems to be working, but it is too early to say if the memory consumption issues are resolved.

I just had to rant about this. I really like wordpress, but i think they missed one really important “usability feature”, take a look at this picture:

Why is the text edior so damn small!? Shouldn’t it be the main part of the “Add new post” page? It only covers a fraction of the screen area. Thank god for firefox extensions like Resizeable Textarea…

Lately I have been experiencing that the mod-mono-server2 process running on the server hosting the video upload web-service (I blogged about this in my last post) has been consuming a lot of memory. Once, it even reached the point where all memory (1.5GB) of the server was consumed and the Linux OOM-killer killed the mod-mono-server2 process. At this point apache failed to restart it. If you are interested you can see the kernel log here and the apache log here. The mono version I am using is mono-apache-server2 1.9.1-2 (Ubuntu 8.10 Intrepid).

I googled the issue and found some information on the mod_mono page:

Under high load, mono process consumes a lot of memory, website stops responding
These symptoms have been reported, but their underlying causes are not known. Set the MonoAutoRestartMode, MonoAutoRestartRequests, MonoMaxActiveRequests, and MonoMaxWaitingRequests directives as described earlier to limit the lifetime of the mono process and to restrict the concurrency happening in the server.

The above describes my issue pretty well, except for the “under high load” part. The video transcoding service is still in beta and is only used by one customer so far. It serves around 500 requests a day, which is not a lot, so I wouldn’t expect this kind of behavior. My guess is that every time a file is uploaded using the UploadFile webmethod additional memory is allocated, but not properly released/reused by mono.

As a workaround for this I will have the mod-mono-server2 process restart restart every 6th hour. This can be done by adding the following lines to /etc/apache2/mods-enabled/mod_mono.conf: Update: This did not work, caused my site to hang entirely every 6th hour.

Hopefully this will limit the memory usage, for now. I would appreciate feedback from anyone who have experienced similar problems on mono.

A few months ago my employer asked me if it would be possible to create a web service for encoding videos. I had been playing around with Amazon’s web services for a while, and it seemed like the perfect foundation for building this.

I decided to build the backend in Python and use ffmpeg for encoding movies. I looked into building the web service frontend in Python as well, but the SOAP libraries I could find for Python did not seem very mature or maintained. Instead I started thinking about building it in ASP.NET (I had previous experiences from building web services in ASP.NET). After some research and testing with Apache and Mono (I wanted to use Linux VMs only) I decided to develop the frontend in ASP.NET but host it on Apache.

To make the service scalable I decided to break it down into several parts and use message passing between the different parts. The parts I broke it down into are:

• Web service frontend – what the user calls to encode a movie. Implemented in ASP.NET, hosted on Apache/Mono on Linux.
• Encode Worker – A python process managing the encoding of videos.
• Encode Master – Manages number of running Encode Workers. Implemented in Python.

When a movie gets uploaded to the web service frontend it gets placed into the encode queue. The encode workers periodically checks if there is anything in the queue, and if it is encodes it. The Encode Master manages the number of running Encode Workers (based on the current length of the encode queue). If the queue growes to long, we just fire up a few new VMs running the worker.

This is a schematic view of how the service has been implemented and how the different components are related to the Amazon services:

Right now we are implementing the solution for our first customer, pretty existing I must say! In an upcoming post I will discuss the different libraries for Python and ASP.NET I used for communicating with the Amazon Web Services.