Sweet Jesus, the Vooddo has been figured out

The one thing that sucks for designy types like me that aren't real programmers is that when ColdFusion switched to Java, a bunch of bearded engineers thought "cool" while people like me when "huh?" The hardest part for me to deal with since the move to java is the tuning of servers. Now I know there is over half a decade of research, tools, and experience for tuning a java server, but that's for the bearded guys. In the coldfusion world, most java stuff is a total mystery, especially the JVM server junk.

So tonight I attended the Portland ColdFusion User Group, which was discussing how to tune the JVM. As everyone knows, MeFi's uptime is a joke and even with forcing service restarts every 12 hours, I'm still seeing the java server kill itself under heavy loads and sometimes up to 5-6 times a day. I've been reading up on jvm tuning, but there's not much out there. There's this document that many people follow, but it doesn't break down each and every setting.

Tonight's talk did. Here are some of my notes:

JVM

  • The latest release from Sun, 1.5, is not supported by Macromedia (you can run it, but no cfdocument, no cfchart tag allowed). 1.42 is the latest supported.
  • basicis of java services: proxy service is the focus
  • scheduler service handling background tasks
  • activeHandlerThreads – maximum number executing (3-5/proc)
  • maxHandlerThreads – active+queued threads set to a large number
  • minHandlerThreads – set near activeHandler Threads?
  • threadwaittimeout – how long to wait in queue
  • threadtimeout – how long until a running proc is killed (30 seconds) set as the same as the CF timeout
  • Memory used by the JVM: young, or eden space for temp variables | tentured or "old" for stuff like application or session variables | perm memory for variables that don't change and garbage collection doesn't run often for

Memory-related JVM Arguments

  • -Xmx: max size of the JVM heap (512Mb is the default)
  • -Xms: initial size of the heap (256Mb is default) could set to max size so that memory never needs to be reallocated
  • -MaxPermSize: size of the perm memory block of JVM (32Mb default)
  • -NewSize: size of the eden memory block of JVM (64Mb default)

Monitoring

  • JVM Stat 3.0
  • visualgc

So the talk walked through using jvmstat 3.0 to monitor memory usage by the JVM. Since MeFi's java server dies every few hours and looks into the logs reveal lots of out of memory errors, I think this is a big part of why the uptime sucks.

Getting jvmstat to work on windows isn't totally easy. You get it here, then follow these instructions to a t, including all the crap about getting your PATH variables right. You also need to install the j2se 5.0 java environment to your server, and if you do that and still get an error that your jdk isn't up to snuff even though it is at 1.5.0, then follow this faq. The last step is to get jvmstat to see your JVM. On windows, this requires shutting down CF App server from the services MMC, then restarting CF using the C:\CfusionMX7\bin\cfstart.bat file. That will run jrun under your user account, and using the task manager you lastly have to find the PID for that process. Pete Frietag has a great post about this.

So the jist of all this junk is that when I finally got jvmstat running, my young, or eden memory was spiking and hitting garbage collection every couple of seconds. When set to 32Mb, it wasn't nearly enough. The Perm memory was barely moving up from the minimum, even though it had a ceiling of 128Mb.

So I did some tweaking. I upped the NewSize from 32Mb to 128Mb. It still peaked under loads, so I tweaked it up to 256Mb. I shrank permsize down to 64Mb max. I also upped the total memory allocation for CFMX to 640Mb, and set the minimum at 512Mb. The web server has 1Gb of ram so with other stuff running, it hovers around 850Mb which is maybe too close for comfort, so I might plunk down the extra $10 a month for another Gb of ram in the box.

For the past few hours, the server is screaming and seems to be responding well to the increased memory footprint and allocation. We'll have to wait and see tomorrow under a normal daytime heavy load if it's enough to keep java running, but I've got a feeling things will be better. But even if it dies a miserable death tomorrow, at least I finally know what all that cryptic voodoo java jvm server settings junk means and I have a tool to track it and examine it so further tweaking will be much easier.

Advertisements