mod-mono-server2 memory consumption problems
Lately I have been experiencing that the mod-mono-server2 process running on the server hosting the video upload web-service (I blogged about this in my last post) has been consuming a lot of memory. Once, it even reached the point where all memory (1.5GB) of the server was consumed and the Linux OOM-killer killed the mod-mono-server2 process. At this point apache failed to restart it. If you are interested you can see the kernel log here and the apache log here. The mono version I am using is mono-apache-server2 1.9.1-2 (Ubuntu 8.10 Intrepid).
I googled the issue and found some information on the mod_mono page:
Under high load, mono process consumes a lot of memory, website stops responding
These symptoms have been reported, but their underlying causes are not known. Set the MonoAutoRestartMode, MonoAutoRestartRequests, MonoMaxActiveRequests, and MonoMaxWaitingRequests directives as described earlier to limit the lifetime of the mono process and to restrict the concurrency happening in the server.
The above describes my issue pretty well, except for the “under high load” part. The video transcoding service is still in beta and is only used by one customer so far. It serves around 500 requests a day, which is not a lot, so I wouldn’t expect this kind of behavior. My guess is that every time a file is uploaded using the UploadFile webmethod additional memory is allocated, but not properly released/reused by mono.
As a workaround for this I will have the mod-mono-server2 process restart restart every 6th hour. This can be done by adding the following lines to /etc/apache2/mods-enabled/mod_mono.conf: Update: This did not work, caused my site to hang entirely every 6th hour.
MonoAutoRestartMode Time
MonoAutoRestartRequests 00:06
Hopefully this will limit the memory usage, for now. I would appreciate feedback from anyone who have experienced similar problems on mono.
I have same problem mono 2.6.1:
1.5GB and server stop responding….
ab -n 100 http://….. => 1.5GB => stop responding
i use directive:
MonoMaxMemory 500000000
and problem solved, ab(apache benchamrk) dont reached 1.5GB… BUT next day server stop responding 2x time, and i must use hard reset… directive work only for one app ? (3x app = 1500GB?) …. this its very bad situation… any solution?
I add now MonoAutoRestartTime 00:01 … :X
Our problem originated in a WebService method that accepted an file upload as byte[]. On Windows using the .NET runtime this did not cause any problem – the memory got freed as supposed. But on Mono the memory allocated for the byte[] (the uploaded file) doesn’t seem to be garbage collected, ever.
Raising the MonoMaxMemory will only delay the problem, once you are out of memory you are out. I really hope the Mono team will adress this issue in the upcoming releases of Mono.
no today webservice called…
and i dont reise maxmemory but limit it to 500MB
How can i limit memory SUM of all solutions to 500MB?
I am note quite sure how to set if for all vhosts, or how well MonoMaxMemory works at all. This is from the documentation:
“MonoMaxMemory. If MonoRunXSP is True, the maximum size of the process’s data segment (data size) in bytes allowed for the spawned mono process. It will be restarted when the limit is reached. If this capability is not available on your system, the directive will be ignored. Default value: system default”
This page states http://www.mono-project.com/Mod_mono:
“Lines 4 and 5 set the maximum memory to be used (bytes) and the maximum CPU time consumed (seconds) by the ‘jeanette’ mod-mono-server instance. After reaching the limit, the OS will kill mod-mono-server. A new instance should start automatically on the next request. (But, JT notes that these directives don’t work at all for him.)”
So it seems like others have problem with the MonoMaxMemory as well…
maybe yesterday really webservice used one of customers.
MonoMaxMemory 500000000 maybe work, but its combination of problems with webservices.
we try dealocation all data manualy…
i past here any news
Hi, did you found how to limit memory consumption of mono runtime ? I have similar problem…
Our solution was to avoid large file uploads to the .NET service and handle them separately by a small PHP application. This did not entirely solve the problem, but at the time is was good enough. I am not working for the company behind this service any more, so I am not sure if this is still the solution they use.
I could recommend looking into using the SGen garbage collector introduced in newer versions of Mono. It could potentially solve the problem.
Why we use mono, with so much issues? I nearly gave up, some services are moved to IIS, and some i am writing new in PHP.
@Saeed Mono itself works fine, but unfortunately the ASP.NET implementation is far from perfect. Also the fastcgi implementation suffers from lack of love. But at least it has gotten better since I wrote this article.
http://sourcecodebean.com/archives/serving-your-asp-net-mvc-site-on-nginx-fastcgi-mono-server4/1617
Personally I run some small trivial applications on Mono or ASP.NET/Mono, but all my production code runs on IIS. Xamarin (the company that now owns the rights and develops Mono) focuses mostly on Mobile development nowadays (MonoTouch and MonoDroid), where they are doing a great job, but I am not sure how much they focus on the web.
Some good news is at least that Mono 2.11.3 was released just a few days ago and includes the newly open sourced Entity Framework.