PerformanceEngineer.com software performance engineering & testing

Finding (and fixing) memory leaks in java can sometimes be a tricky process. I recently had to do some analysis on a leaking application and hit a snag.

My game plan was to do the following:

• Capture heap dumps with jmap
• Analyze dump with jhat

Along the way, I ran into a problem with jmap and discovered a couple new tools.

Problem with jmap

The first problem I had was actually generating the heap dumps. My application was running 32 bit JVM version 1.5.0_11-b03 on Solaris 10, and every time I tried running

jmap -heap:format=b $pid

, the process would start, but never finish. I could see that the heap.bin file was created and growing in size, but the growth rate would get slower and slower until, eventually, it would fail with:

Exception in thread "main" java.lang.OutOfMemoryError: 
    requested 8192 bytes for jbyte in 
/BUILD_AREA/jdk1.5.0_11/hotspot/src/share/vm/prims/jni.cpp. Out of swap space?

Well, after multiple attempts and trying to run on machines with more than 2 GB of swap space, I started to monitor the jmap process and found that it would grow and grow in size, until almost 4 GB when it would fail with the OOME. Ok, now this is looking like a bug. I found a similar bug for an older version of the JDK, but the symptoms were similar.

After installing JDK 1.5.0_15, I tried again and bingo, worked like a charm. Damn, wasted a good few hours there.

Next... Analysis

I had originally planned on doing all of the leak analysis using jhat, but I stumbled upon the SAP Memory Analyzer while researching the jmap problem and boy, am I glad I did.

It looks like jhat has potential to be quite powerful with OQL (object query language), but it was terribly slow with my 600 MB heap dumps. Although jhat is distributed with JDK 1.6, it can handle heap dumps created in 1.5 with no problem.

In contrast, the SAP Memory Analyzer was fantastic. It had no problem with large heap dumps, it was fast and it led me almost directly to the leaking culprits. The documentation provided was also very helpful.

Conclusion

In summary, the process works like this:

1. Capture Heap Dump
2. Analyze Heap Dump

For Step 1, you have option of using jmap against a running process or a core file, which can be created using gcore. The syntax is slightly different depending on your JDK version (see Resources below).

For Step 2, I would highly recommend the SAP Memory Analyzer. I actually switched back and forth between it and jhat, but if you were going to choose just one, go with the SAP tool.

Resources

jmap - JDK 1.5
jmap - JDK 1.6
jhat - Java Heap Analysis Tool
HAT Java 1.5 Heap Analysis Tool
SAP Memory Analyzer
Edward Chou's Blog - provides some good jhat tips