I missed the memory talk at VMWorld. I passed it by for some API related sessions. One of my coworkers suggested I go to it and I should have listened to him.
Something that’s popped up recently is the fact that memory reclamation using the balloon driver fails miserably when used in conjunction with a JVM.
- http://communities.vmware.com/blogs/drummonds/2009/09/09/love-your-balloon-driver
- http://www.vmware.com/files/pdf/perf-vsphere-memory_management.pdf
These articles put the blame squarely on Java's garbage collection. The latter pointing out that the garbage collect will scan all the process's memory making paging a JVM heap worse than paging most applications.
I'd like to see some poeple that know more about JVM memory internals and garbage collection as well as VMWare weigh in on this as well. With virtual machines being the basis of cloud initiatives, and future collocation platforms, this kind of issue is only going to get more prevalent.
Currently, the only solution is have admins take a closer role in managing the JVM memory footprint at the same time as the VMWare VM memory reservations.
I'm not a fan of the maximum heap size setting in the JVM either. But there are arguments for and against it being require in the JVMs style of garbage collection.
I'm the dev lead on the EM4J project that was spoken about at VMWorld. You're right to surmise that the JVM has a problem with regular ballooning. At it's most simple, the problem is that the JVM doesn't give memory back to the operating system and thus always ends up consuming it's high watermark memory, even if there's plenty of free space in the heap. Since the balloon driver can only reclaim memory from the OS, these two models are basically incompatible. The result is that it's fiendishly difficult to predict when regular ballooning is safe and the consequences of getting it wrong are severe (due to GC characteristics as you state). Hence the current best practice of using memory reservations.
I'm putting together a series of YouTube clips to explain this whole area in more detail. Hopefully these will be helpful. First one is up now: http://www.youtube.com/watch?v=kyz7J-FQUSM