Feedlr has been running quite well for a while, until a couple weeks ago, the server started to choke a lot. And I realized I had to do something to improve the performance. Actually, until then I had barely paid any attention performance-wise - getting the app up and running was the goal. And then with the growth of the app it’s time to do some work for that.
The Problem
The symptom of the performance bottleneck was that from time to time Tomcat would take a long time to respond, resulting in timeouts at the browser clients. This is caused by heavy swapping - apparently the 512MB RAM shared by all the services on the VPS, including Apache (with worker mpm), php, MySQL, and Tomcat, is less than abundant. But before upgrading my VPS, I want to see how far I can go with the current specs.
Attempting to optimize the code
My first attempt was to review my code and try to identify any possible memory leaks and memory inefficient spots. After working with profiling tools (I used an evaluation version of JProfiler, which is very good), and several refractory efforts, there was no obvious improvement. It didn’t look like a memory leak. And the effort trying to squeeze more memory from the code didn’t seem to pay off.
Tuning the JVM
Then I began to focus on tuning the JVM. The sporadic heavy swapping hints at inefficient garbage collections. But why is GC taking so much resources? Using GCViewer, I can see that the throughput can be as low as around 70%. And full GC time constantly reaches several minutes. (SliceHost has sent me a dozen heavy swap usage report already)
After some searching and study, first thing I tried was to decrease the maximum heap size. Yes, decrease it. Previously, I was assigning 384MB to the heap. Because while Tomcat is not running, I have a little less than 440MB free physical memory. Leaving some buffer, 384MB seems a reasonable number. But the excessive swapping and very long full GC time indicates that part of the heap must be slipping into the virtual memory. And virtual memory is the biggest enemy of JVM garbage collection. So I decreased the heap size incrementally, arriving at 270MB. And the GC log shows that the time full GC takes has considerably dropped.
To further improve the situation, I tried different GC policies. First I tried the Concurrent Mark Sweep GC policy, which is supposed to work very well with multi-processor servers and is optimized for response time. Profiling with the CMS options on my local workstation shows good results. And since “cat /proc/cpuinfo” shows that my VPS is equipped with 4 CPU’s, I was expecting to see obvious improvements.
However, to my great disappointment, CMS totally strangled my VPS. Soon after Tomcat starts up, CPU usage became constantly high and the Grails app responds even slower! I’m not exactly sure why, but it looks like the number of CPU’s shown at your VPS doesn’t mean it has the comparable concurrent processing power.
Then I gave up the CMS policy and tried the other GC policy suitable for servers, the Parallel policy. This turned out to work quite well on the VPS, together with “-XX:+UseParallelOldGC” which is available since Java 5 update 6 and enhanced in Java 6 which I’m running my app on.
The Results
I experimented with a few other JVM options, and finally achieved a throughput of above 98%. Full GC now usually takes around 1 second, occasionally maxing at over 10 seconds. I think there are some hard limitations to the server, but the results are already good enough for now. Here are the current ones which have achieved reasonable performance for my Grails app. These may not be the perfect setup since I haven’t tried the combinations exhaustively.
-server -Xmx270m -Xms270m -XX:MaxPermSize=80m -Xverify:none -XX:+UseParallelGC -XX:+UseParallelOldGC -XX:+UseAdaptiveSizePolicy -XX:SurvivorRatio=4 -XX:TargetSurvivorRatio=90 -XX:MaxTenuringThreshold=31 -XX:+AggressiveOpts
References
Below are some useful articles I’ve come across while tuning the JVM.
Further thoughts
I really love Grails for what it is: remarkably agile development with real joy. But as the person running the app, I have to deal with various issues of the stack that Grails sits on, which sometimes frustrates me.
On one hand this robust stack really grants Grails an advantage in the enterprise market. But on the other hand, this is no good news for the creators of hundreds and thousands of small web apps, pet projects, interesting weekend hacks and alike. The most obvious obstacle is Java hosting. It’s 2008 already, and there are still simply _no_ easily affordable and good enough hosting services for the Java stack! The entry cost is too high. With innovative services like Heroku and Googel App Engine emerging, the gap is even increasing. The barrier to actually making something useful on the web is being dramatically lowered. But yet, the Java stack is left out in the cold.
While this is a problem, it’s also an incredible opportunity. With Grails emerging, the Java stereotype can be changed. People can build agile web apps on top of the proven stack, which is the core competency of Grails as I see it. I want to build things with Grails. But I don’t want to worry about how to host my app. Now, who’s gonna give us the ultimate service to grow little ideas into reality?
