source code bean » Mono http://sourcecodebean.com thoughts and ideas from a .net developer Fri, 27 Jan 2012 20:36:28 +0000 en hourly 1 http://wordpress.org/?v=3.3.1 MongoDB, C# and Mono http://sourcecodebean.com/archives/mongodb-c-and-mono/1408 http://sourcecodebean.com/archives/mongodb-c-and-mono/1408#comments Wed, 11 Jan 2012 18:53:48 +0000 Peter http://sourcecodebean.com/?p=1408 MongoDB is one of the many free open source NoSQL databases that exist today. I wanted to try out how well the official drivers for C# worked when using Mono.

On the MongoDB web site they have pre built binaries of MongoDB for almost every platform, i grabbed the 64 bit binary for OSX. No installation is required, just unzip the file and run the “mongod” binary from the directory. There is also a binary called “mongo”, this is the mongo interactive shell which is very useful.

The C# drivers are available, both in binary form and as source, from the C# language center. To get started I fired up MonoDevelop 2.8 running on Mono 2.10.8:



MonoDevelop has matured a lot since I used it the first time. It is still not as good as Visual Studio, but it is getting there. What takes most time to get used to is the totally different set of shortcuts, and the lack of all ReSharper magic.

Let me break down a simple console application example on how to use MongoDB using the official driver from 10gen:

  1.  
  2.  
  3. using System;
  4. using System.Collections.Generic;
  5. using System.Linq;
  6. using MongoDB.Bson;
  7. using MongoDB.Driver;
  8. using MongoDB.Driver.Builders;
  9.  
  10. namespace MongoDbTest
  11. {
  12.   public class Article
  13.   {
  14.     public MongoDB.Bson.ObjectId id { get; set; }
  15.     public string title { get; set; }
  16.     public string status { get; set; }
  17.  
  18.     public string[] tags { get; set; } 
  19.   }
  20.        
  21.   class MainClass
  22.   {
  23.     public static void Main (string[] args)
  24.     {
  25.       // Connect to the database and get a colleciton
  26.       MongoServer server = MongoServer.Create ("mongodb://localhost");
  27.       MongoDatabase db = server.GetDatabase ("mongodb-tutorial");
  28.                        
  29.       // Get an article collection
  30.       MongoCollection<BsonDocument> articles = db.GetCollection<BsonDocument> ("articles");
  31.                        
  32.       // Empty the collection (so all runs of this program will result in the same output)
  33.       articles.RemoveAll ();
  34.  
  35.  
  36.       //               
  37.       // Create two articles
  38.       //
  39.       articles.Save (new BsonDocument
  40.       {
  41.         {"title", "A MongoDB article"},
  42.         {"status", "published"},
  43.         {"tags", new BsonArray { "c#", "mongodb" }}     
  44.       });
  45.                        
  46.       articles.Save (new BsonDocument
  47.       {
  48.         {"title", "A Mono article"},
  49.         {"status", "draft"},
  50.         {"tags", new BsonArray { "c#", "mono" }}       
  51.       });              
  52.  

If you type this in the MongoDB shell, you will see that the following articles are stored in the database:

> db.articles.find()
        {
          "_id" : ObjectId("4f097f8e74b5b343b97f56a7"),
          "title" : "A MongoDB article",
          "status" : "published",
          "tags" : [ "c#", "mongodb" ]
        }

        {
          "_id" : ObjectId("4f097f8e74b5b343b97f56a8"),
          "title" :
          "A Mono article",
          "status" : "draft",
          "tags" : [ "c#", "mono" ]
       }
  1.        
  2.      // Find and print the first article
  3.       BsonDocument firstArticle = articles.FindOne ();
  4.      
  5.       PrintArticle ( // Access the properties using indexing and then casting using the .As*
  6.         firstArticle["_id"].AsObjectId,
  7.         firstArticle["title"].AsString,
  8.         firstArticle["status"].AsString,
  9.         (firstArticle["tags"].AsBsonArray).Select (x => x.AsString));          
  10.      
  11.  
  12.       //
  13.       // Typed collections                     
  14.       //
  15.  
  16.       // Using indexing and the .As* operations is not that convenient. A better way
  17.       // is to use a typed collection. Earlier we defined a Article class, lets use it:
  18.       MongoCollection<Article> typedArticles = db.GetCollection<Article> ("articles");
  19.       Article typedArticle = typedArticles.FindOneAs<Article> ();
  20.      
  21.       PrintArticle (typedArticle.id, typedArticle.title,
  22.         typedArticle.status, typedArticle.tags);
  23.                        
  24.       // Save a typed article
  25.       typedArticles.Save (new Article {
  26.                                 title = "An article about MonoDevelop",
  27.                                 status = "published",
  28.                                 tags = new [] {"c#", "mono", "mono-develop"},
  29.                         });
  30.  
  31.  
  32.       //
  33.       // Querying
  34.       //
  35.                
  36.       // Find article with status draft
  37.       Article draftArticle = typedArticles
  38.         .FindOneAs<Article> (Query.EQ ("status", "draft"));
  39.      
  40.       PrintArticle (draftArticle.id, draftArticle.title,
  41.         draftArticle.status, draftArticle.tags);
  42.                        
  43.  
  44.       //
  45.       // Updating
  46.       //
  47.                
  48.       // Lets update the draft article and set status published
  49.       Console.WriteLine ("Nr of published articles before update:"
  50.                          + typedArticles.Find (Query.EQ ("status", "published")).Count ());
  51.                          
  52.       typedArticles.Update (
  53.         new QueryDocument { { "_id", draftArticle.id } },
  54.         new UpdateDocument { { "$set", new BsonDocument ("status", "published") }
  55.         // $set is atomic
  56.       });
  57.                
  58.       Console.WriteLine ("Nr of pubhlished articles after update:"
  59.                          + typedArticles.Find (Query.EQ ("status", "published")).Count ());
  60.                
  61.      
  62.       //
  63.       // MapReduce
  64.       //
  65.                
  66.       // This will count the number of times a tag has been used by using MapReduce
  67.       var map =
  68.         "function() {" +
  69.         "    this.tags.forEach(function(t) {" + // Iterate all the tags and
  70.         "        emit(t, { count : 1 }); " +    // emit the tag name and initial count
  71.         "    })" +
  72.         "}";
  73.                        
  74.       var reduce =
  75.         "function(key, emits) {" + // Reduce by tag name and summarize the result
  76.         "    total = 0;" +
  77.         "    for (var i in emits) {" +
  78.         "        total += emits[i].count;" +
  79.         "    }" +
  80.         "    return { count : total };" +
  81.         "}";
  82.  
  83.  
  84.       Console.WriteLine ("MapReduce result:");         
  85.       var mr = typedArticles.MapReduce (map, reduce);
  86.       foreach (var document in mr.GetResults()) {
  87.         Console.WriteLine (document.ToJson ());
  88.       }                
  89.     }
  90.                
  91.     public static void PrintArticle (ObjectId id, string title,
  92.       string status, IEnumerable<string> tags)
  93.     {
  94.       Console.WriteLine ("id: " + id + ", title: " + title);
  95.       Console.WriteLine ("- tags: " + string.Join (", ", tags));
  96.     }
  97.   }
  98. }
  99.  

Output from the program:

id: 4f0989fc74b5b3458ae38105, title: A MongoDB article
- tags: c#, mongodb
id: 4f0989fc74b5b3458ae38105, title: A MongoDB article
- tags: c#, mongodb
id: 4f0989fc74b5b3458ae38106, title: A Mono article
- tags: c#, mono
Nr of pubhlished articles before update:2
Nr of pubhlished articles after update:3
MapReduce result:
{ “_id” : “c#”, “value” : { “count” : 3.0 } }
{ “_id” : “mongodb”, “value” : { “count” : 1.0 } }
{ “_id” : “mono”, “value” : { “count” : 2.0 } }
{ “_id” : “mono-develop”, “value” : { “count” : 1.0 } }

Using the C# driver for MongoDB, is as you can see, very straight forward. Usually when I run applications on Mono I make sure to build all libraries using the Mono compiler, to make sure the Microsoft compiler have not used any optimizations that is not supported under Mono (I have run into this issue in the past). However, in this case the .NET binaries on the MongoDB site seems to work just fine under Mono. I also downloaded the MongoDB C# driver source and build them using MonoDevelop without any problems.

One improvement I would like to see is LINQ support in the driver. It would be so much more convenient to be able to use LINQ and strongly typed objects to query the database or create MapReduce-functions.

]]> http://sourcecodebean.com/archives/mongodb-c-and-mono/1408/feed 0 An update on the mono memory issue http://sourcecodebean.com/archives/an-update-on-the-mono-memory-issue/184 http://sourcecodebean.com/archives/an-update-on-the-mono-memory-issue/184#comments Wed, 13 May 2009 13:21:16 +0000 Peter http://sourcecodebean.com/?p=184 The workaround I tested a few weeks ago did not really solve my problem – actually it made it worse. The autorestart caused mono to hang and not restart at all, so my site stopped responding every 6th hour. I quickly had to disable this. Still I had the memory consumption problems. From various sources I was told that the memory issues would be fixed in the recently released mono 2.4. Also I found a bug report regarding AutoRestart, which also should have been fixed in 2.4. So I decided to give Mono 2.4 a try.
The problem was that there were no packages for Ubuntu 8.10, so I had to download the sources and build my own deb packages. I found this great blog post that describes the process of building and installing Mono 2.4.

So far it seems to be working, but it is too early to say if the memory consumption issues are resolved.

]]> http://sourcecodebean.com/archives/an-update-on-the-mono-memory-issue/184/feed 2 mod-mono-server2 memory consumption problems http://sourcecodebean.com/archives/mod-mono-server2-memory-consumption-problems/121 http://sourcecodebean.com/archives/mod-mono-server2-memory-consumption-problems/121#comments Tue, 31 Mar 2009 21:21:49 +0000 Peter http://sourcecodebean.com/?p=121 Lately I have been experiencing that the mod-mono-server2 process running on the server hosting the video upload web-service (I blogged about this in my last post) has been consuming a lot of memory. Once, it even reached the point where all memory (1.5GB) of the server was consumed and the Linux OOM-killer killed the mod-mono-server2 process. At this point apache failed to restart it. If you are interested you can see the kernel log here and the apache log here. The mono version I am using is mono-apache-server2 1.9.1-2 (Ubuntu 8.10 Intrepid).

I googled the issue and found some information on the mod_mono page:

Under high load, mono process consumes a lot of memory, website stops responding
These symptoms have been reported, but their underlying causes are not known. Set the MonoAutoRestartMode, MonoAutoRestartRequests, MonoMaxActiveRequests, and MonoMaxWaitingRequests directives as described earlier to limit the lifetime of the mono process and to restrict the concurrency happening in the server.

The above describes my issue pretty well, except for the “under high load” part. The video transcoding service is still in beta and is only used by one customer so far. It serves around 500 requests a day, which is not a lot, so I wouldn’t expect this kind of behavior. My guess is that every time a file is uploaded using the UploadFile webmethod additional memory is allocated, but not properly released/reused by mono.

As a workaround for this I will have the mod-mono-server2 process restart restart every 6th hour. This can be done by adding the following lines to /etc/apache2/mods-enabled/mod_mono.conf: Update: This did not work, caused my site to hang entirely every 6th hour.

  1. MonoAutoRestartMode Time
  2. MonoAutoRestartRequests 00:06

Hopefully this will limit the memory usage, for now. I would appreciate feedback from anyone who have experienced similar problems on mono.

]]> http://sourcecodebean.com/archives/mod-mono-server2-memory-consumption-problems/121/feed 6 Building a distributed web service using Amazon Web Services http://sourcecodebean.com/archives/building-a-distributed-web-service-using-amazon-web-services/93 http://sourcecodebean.com/archives/building-a-distributed-web-service-using-amazon-web-services/93#comments Sun, 22 Mar 2009 18:18:08 +0000 Peter http://sourcecodebean.com/?p=93 A few months ago my employer asked me if it would be possible to create a web service for encoding videos. I had been playing around with Amazon’s web services for a while, and it seemed like the perfect foundation for building this.

I decided to build the backend in Python and use ffmpeg for encoding movies. I looked into building the web service frontend in Python as well, but the SOAP libraries I could find for Python did not seem very mature or maintained. Instead I started thinking about building it in ASP.NET (I had previous experiences from building web services in ASP.NET). After some research and testing with Apache and Mono (I wanted to use Linux VMs only) I decided to develop the frontend in ASP.NET but host it on Apache.


To make the service scalable I decided to break it down into several parts and use message passing between the different parts. The parts I broke it down into are:

  • Web service frontend – what the user calls to encode a movie. Implemented in ASP.NET, hosted on Apache/Mono on Linux.
  • Encode Worker – A python process managing the encoding of videos.
  • Encode Master – Manages number of running Encode Workers. Implemented in Python.

When a movie gets uploaded to the web service frontend it gets placed into the encode queue. The encode workers periodically checks if there is anything in the queue, and if it is encodes it. The Encode Master manages the number of running Encode Workers (based on the current length of the encode queue). If the queue growes to long, we just fire up a few new VMs running the worker.

This is a schematic view of how the service has been implemented and how the different components are related to the Amazon services:

Video Encoding Service

Right now we are implementing the solution for our first customer, pretty existing I must say! In an upcoming post I will discuss the different libraries for Python and ASP.NET I used for communicating with the Amazon Web Services.

]]> http://sourcecodebean.com/archives/building-a-distributed-web-service-using-amazon-web-services/93/feed 2