source code bean » Mono

An update on the mono memory issue

Peter — Wed, 13 May 2009 13:21:16 +0000

The workaround I tested a few weeks ago did not really solve my problem – actually it made it worse. The autorestart caused mono to hang and not restart at all, so my site stopped responding every 6th hour. I quickly had to disable this. Still I had the memory consumption problems. From various sources I was told that the memory issues would be fixed in the recently released mono 2.4. Also I found a bug report regarding AutoRestart, which also should have been fixed in 2.4. So I decided to give Mono 2.4 a try.
The problem was that there were no packages for Ubuntu 8.10, so I had to download the sources and build my own deb packages. I found this great blog post that describes the process of building and installing Mono 2.4.

So far it seems to be working, but it is too early to say if the memory consumption issues are resolved.

mod-mono-server2 memory consumption problems

Peter — Tue, 31 Mar 2009 21:21:49 +0000

Lately I have been experiencing that the mod-mono-server2 process running on the server hosting the video upload web-service (I blogged about this in my last post) has been consuming a lot of memory. Once, it even reached the point where all memory (1.5GB) of the server was consumed and the Linux OOM-killer killed the mod-mono-server2 process. At this point apache failed to restart it. If you are interested you can see the kernel log here and the apache log here. The mono version I am using is mono-apache-server2 1.9.1-2 (Ubuntu 8.10 Intrepid).

I googled the issue and found some information on the mod_mono page:

Under high load, mono process consumes a lot of memory, website stops responding
These symptoms have been reported, but their underlying causes are not known. Set the MonoAutoRestartMode, MonoAutoRestartRequests, MonoMaxActiveRequests, and MonoMaxWaitingRequests directives as described earlier to limit the lifetime of the mono process and to restrict the concurrency happening in the server.

The above describes my issue pretty well, except for the “under high load” part. The video transcoding service is still in beta and is only used by one customer so far. It serves around 500 requests a day, which is not a lot, so I wouldn’t expect this kind of behavior. My guess is that every time a file is uploaded using the UploadFile webmethod additional memory is allocated, but not properly released/reused by mono.

~~As a workaround for this I will have the mod-mono-server2 process restart restart every 6th hour. This can be done by adding the following lines to /etc/apache2/mods-enabled/mod_mono.conf:~~ Update: This did not work, caused my site to hang entirely every 6th hour.

MonoAutoRestartMode Time
MonoAutoRestartRequests 00:06

Hopefully this will limit the memory usage, for now. I would appreciate feedback from anyone who have experienced similar problems on mono.

Building a distributed web service using Amazon Web Services

Peter — Sun, 22 Mar 2009 18:18:08 +0000

A few months ago my employer asked me if it would be possible to create a web service for encoding videos. I had been playing around with Amazon’s web services for a while, and it seemed like the perfect foundation for building this.

I decided to build the backend in Python and use ffmpeg for encoding movies. I looked into building the web service frontend in Python as well, but the SOAP libraries I could find for Python did not seem very mature or maintained. Instead I started thinking about building it in ASP.NET (I had previous experiences from building web services in ASP.NET). After some research and testing with Apache and Mono (I wanted to use Linux VMs only) I decided to develop the frontend in ASP.NET but host it on Apache.

To make the service scalable I decided to break it down into several parts and use message passing between the different parts. The parts I broke it down into are:

Web service frontend – what the user calls to encode a movie. Implemented in ASP.NET, hosted on Apache/Mono on Linux.
Encode Worker – A python process managing the encoding of videos.
Encode Master – Manages number of running Encode Workers. Implemented in Python.

When a movie gets uploaded to the web service frontend it gets placed into the encode queue. The encode workers periodically checks if there is anything in the queue, and if it is encodes it. The Encode Master manages the number of running Encode Workers (based on the current length of the encode queue). If the queue growes to long, we just fire up a few new VMs running the worker.

This is a schematic view of how the service has been implemented and how the different components are related to the Amazon services:

Right now we are implementing the solution for our first customer, pretty existing I must say! In an upcoming post I will discuss the different libraries for Python and ASP.NET I used for communicating with the Amazon Web Services.