source code bean » NoSQL

MongoDB, C# and Mono

Peter — Wed, 11 Jan 2012 18:53:48 +0000

MongoDB is one of the many free open source NoSQL databases that exist today. I wanted to try out how well the official drivers for C# worked when using Mono.

On the MongoDB web site they have pre built binaries of MongoDB for almost every platform, i grabbed the 64 bit binary for OSX. No installation is required, just unzip the file and run the “mongod” binary from the directory. There is also a binary called “mongo”, this is the mongo interactive shell which is very useful.

The C# drivers are available, both in binary form and as source, from the C# language center. To get started I fired up MonoDevelop 2.8 running on Mono 2.10.8:

MonoDevelop has matured a lot since I used it the first time. It is still not as good as Visual Studio, but it is getting there. What takes most time to get used to is the totally different set of shortcuts, and the lack of all ReSharper magic.

Let me break down a simple console application example on how to use MongoDB using the official driver from 10gen:

using System;
using System.Collections.Generic;
using System.Linq;
using MongoDB.Bson;
using MongoDB.Driver;
using MongoDB.Driver.Builders;
namespace MongoDbTest
{
public class Article
{
public MongoDB.Bson.ObjectId id { get; set; }
public string title { get; set; }
public string status { get; set; }
public string[] tags { get; set; }
}
class MainClass
{
public static void Main (string[] args)
{
// Connect to the database and get a colleciton
MongoServer server = MongoServer.Create ("mongodb://localhost");
MongoDatabase db = server.GetDatabase ("mongodb-tutorial");
// Get an article collection
MongoCollection articles = db.GetCollection ("articles");
// Empty the collection (so all runs of this program will result in the same output)
articles.RemoveAll ();
//
// Create two articles
//
articles.Save (new BsonDocument
{
{"title", "A MongoDB article"},
{"status", "published"},
{"tags", new BsonArray { "c#", "mongodb" }}
});
articles.Save (new BsonDocument
{
{"title", "A Mono article"},
{"status", "draft"},
{"tags", new BsonArray { "c#", "mono" }}
});

If you type this in the MongoDB shell, you will see that the following articles are stored in the database:

> db.articles.find()
        {
          "_id" : ObjectId("4f097f8e74b5b343b97f56a7"),
          "title" : "A MongoDB article",
          "status" : "published",
          "tags" : [ "c#", "mongodb" ]
        }

        {
          "_id" : ObjectId("4f097f8e74b5b343b97f56a8"),
          "title" :
          "A Mono article",
          "status" : "draft",
          "tags" : [ "c#", "mono" ]
       }

// Find and print the first article
BsonDocument firstArticle = articles.FindOne ();
PrintArticle ( // Access the properties using indexing and then casting using the .As*
firstArticle["_id"].AsObjectId,
firstArticle["title"].AsString,
firstArticle["status"].AsString,
(firstArticle["tags"].AsBsonArray).Select (x => x.AsString));
//
// Typed collections
//
// Using indexing and the .As* operations is not that convenient. A better way
// is to use a typed collection. Earlier we defined a Article class, lets use it:
MongoCollection
typedArticles = db.GetCollection
("articles");
Article typedArticle = typedArticles.FindOneAs
();
PrintArticle (typedArticle.id, typedArticle.title,
typedArticle.status, typedArticle.tags);
// Save a typed article
typedArticles.Save (new Article {
title = "An article about MonoDevelop",
status = "published",
tags = new [] {"c#", "mono", "mono-develop"},
});
//
// Querying
//
// Find article with status draft
Article draftArticle = typedArticles
.FindOneAs
(Query.EQ ("status", "draft"));
PrintArticle (draftArticle.id, draftArticle.title,
draftArticle.status, draftArticle.tags);
//
// Updating
//
// Lets update the draft article and set status published
Console.WriteLine ("Nr of published articles before update:"
+ typedArticles.Find (Query.EQ ("status", "published")).Count ());
typedArticles.Update (
new QueryDocument { { "_id", draftArticle.id } },
new UpdateDocument { { "$set", new BsonDocument ("status", "published") }
// $set is atomic
});
Console.WriteLine ("Nr of pubhlished articles after update:"
+ typedArticles.Find (Query.EQ ("status", "published")).Count ());
//
// MapReduce
//
// This will count the number of times a tag has been used by using MapReduce
var map =
"function() {" +
" this.tags.forEach(function(t) {" + // Iterate all the tags and
" emit(t, { count : 1 }); " + // emit the tag name and initial count
" })" +
"}";
var reduce =
"function(key, emits) {" + // Reduce by tag name and summarize the result
" total = 0;" +
" for (var i in emits) {" +
" total += emits[i].count;" +
" }" +
" return { count : total };" +
"}";
Console.WriteLine ("MapReduce result:");
var mr = typedArticles.MapReduce (map, reduce);
foreach (var document in mr.GetResults()) {
Console.WriteLine (document.ToJson ());
}
}
public static void PrintArticle (ObjectId id, string title,
string status, IEnumerable tags)
{
Console.WriteLine ("id: " + id + ", title: " + title);
Console.WriteLine ("- tags: " + string.Join (", ", tags));
}
}
}

Output from the program:

id: 4f0989fc74b5b3458ae38105, title: A MongoDB article
- tags: c#, mongodb
id: 4f0989fc74b5b3458ae38105, title: A MongoDB article
- tags: c#, mongodb
id: 4f0989fc74b5b3458ae38106, title: A Mono article
- tags: c#, mono
Nr of pubhlished articles before update:2
Nr of pubhlished articles after update:3
MapReduce result:
{ “_id” : “c#”, “value” : { “count” : 3.0 } }
{ “_id” : “mongodb”, “value” : { “count” : 1.0 } }
{ “_id” : “mono”, “value” : { “count” : 2.0 } }
{ “_id” : “mono-develop”, “value” : { “count” : 1.0 } }

Using the C# driver for MongoDB, is as you can see, very straight forward. Usually when I run applications on Mono I make sure to build all libraries using the Mono compiler, to make sure the Microsoft compiler have not used any optimizations that is not supported under Mono (I have run into this issue in the past). However, in this case the .NET binaries on the MongoDB site seems to work just fine under Mono. I also downloaded the MongoDB C# driver source and build them using MonoDevelop without any problems.

One improvement I would like to see is LINQ support in the driver. It would be so much more convenient to be able to use LINQ and strongly typed objects to query the database or create MapReduce-functions.

Trying out CouchDB for the first time

Peter — Mon, 08 Mar 2010 06:05:59 +0000

CouchDB is a document-oriented database written in the Erlang programming language. It is a NoSQL product designed for local replication and to scale vertically along a wide range of devices.

There are several good libraries that will abstract accessing CouchDB from various programing languages. However, in order to understand what goes on in the libraries, I think it is important to first understand what is going on on a lower lever, so that is what I will show you.

Step 1, install CouchDB. CouchDB for OSX can be downloaded here, the bundle contains bouth the Erlang runtime and CouchDB – no compiling or installing needed, just download the .dmg, mount it and run the application! If you are on Linux it is quite likely that CouchDB is in the repository of your distribution. In Ubuntu CouchDB is found in “Universe” and can easily be installed using apt. I haven’t tried CouchDB on Windows so I can’t give you any guidance here, if you try it out, please leave a comment.

Now when CouchDB is installed lets explore it using use one of my favorite tools, good old cURL. Port 5984 is the default port for CouchDB.

$ curl -X GET http://localhost:5984/
{"couchdb":"Welcome","version":"0.10.0"}

CouchDB is up running! Lets create a new database:

$ curl -X PUT http://localhost:5984/testdb
{"ok":true}

CouchDB’s responses are also in JSON form. Lets inspect the database we just created:

$ curl -X GET http://localhost:5984/testdb
{
"db_name":"testdb",
"doc_count":0,
"doc_del_count":0,
"update_seq":0,
"purge_seq":0,
"compact_running":false,
"disk_size":79,
"instance_start_time":"1266963717052501",
"disk_format_version":4
}

CouchDB returns some statistics of the database, we can see that it contains 0 documents. Lets store an empty document in the database:

$ curl -X POST http://localhost:5984/testdb/ -H "Content-Type: application/json" -d {}
{
"ok":true,
"id":"ef40feff87010a6ef3a45a16df5af977",
"rev":"1-967a00dff5e02add41819138abb3284d"
}

CouchDB returns the unique id for the document and the version number (did i mention that all documents are version controlled?:)). Next step is to fetch all documents:

$ curl -X GET http://localhost:5984/testdb/_all_docs
{
"total_rows":1,
"offset":0,
"rows":[{
"id":"ef40feff87010a6ef3a45a16df5af977",
"key":"ef40feff87010a6ef3a45a16df5af977",
"value":{"rev":"1-967a00dff5e02add41819138abb3284d"}
}]
}

Yes! it is there! Now you should also be able to see your database “testdb” and the document you created in the CouchDB GUI.

CouchDB a NoSQL database

Peter — Mon, 01 Mar 2010 20:44:29 +0000

Traditionally relational databases has been the primary way of storing, sorting and searching data, and for most purposes they are very good at it. However, in the last few years – with the growth of cloud computing and sites such as Facebook pushing relational databases to its limits, people have started to look for other alternatives. According to some people the problems with relational databases is that it has been hard to scale them in a vertical way (scale them over several servers with linear, or close to linear, performance increase). A few databases in a cluster is no problem, but when it comes to several terabytes of data, which should be searched in real time, they are simply not enough. CouchDB and the other NoSQL databases aims to provide a truly horizontally scalable database.

This opinion that relational databases databases can not scale in the same way is not shared by everyone, a good read about this is Dennis Forbes Getting Real about NoSQL and the SQL-Isn’t-Scalable Lie. However, I think there is definitely a place where NoSQL can prove to be very usable, and it is an interesting technology, so I will let the experts fight about this and instead give you more insight into what NoSQL really is all about.

NoSQL is a umbrella term for a wide variety of data stores, which all have in common that they do not store data in a relational way. Some examples of NoSQL databases are CouchDB, MongoDB, Amazon SimpleDB and Google BigTable. This is some of the properties they have in common:

1. No schema
The data is stored in one big hashtable like data structure. No schema is needed.

2. No more joins
Joins are slow in general, when they are spread out over several servers it gets even worse. In CouchDB there is no join, instead data should be duplicated. This might sound odd if you like me was taucht back in collage that normalization is of highest importance, and that you should really really avoid to duplicate data in your database. This is still true for most relational databases, but keep in mind that when relational databases was invented, disk space was expensive and normalization was a great way to save a few bytes. Storage is everything but expensive today, this is why NoSQL empathizes that data should instead be duplicated.

3. Eventual consistency
When you update the database there is no longer a quarantine that all subsequent queries will get the updated value immediately, it might take some time depending on the system load. For some systems such as banking systems, this kind of behavior would be a big no no. But for most large web sites, this is no problem, the data is going to be cached in one way or another anyway.

CouchDB

CouchDB is a document database, accessible via a RESTful JSON API. Everything stored in the database is a “document” and is stored in a flat address space.There are no schemas, the documents are stored and retrieved as JSON objects.

To address this problem of adding structure back to semi-structured data, CouchDB has something called “views”. A view is as close as we gets to a SQL query. The views are expressed in Javascript and consist of map and filter functions. They are built dynamically and will not affect the underlying data.

CouchDB is written in Erlang and runs on all systems that the Erlang runtime supports (Linux, Windows, OSX and other unix systems). I have tested CouchDB on Linux and OSX. This is a screenshot of the CouchDBX GUI running on OSX:

The JSON representation of the same document:

In my next post I will show you some hands on action with CouchDB.