Bottlenecks: Part 2 of 4
November 14, 2007 – 10:12 amThis is the second in a four part series about finding bottlenecks in an application or system.
Memory Bottlenecks
A memory bottleneck is a condition where a lack of memory (or general limitation of memory access) slows the performance or scalability of the application. I will discuss how to detect two common scenarios of memory bottlenecks, there are certainly more possibilities depending on your application and platform.
Physical Memory
On UNIX, a memory bottleneck can be detected easily using the ‘vmstat’ utility. Contrary to popular belief, the presence of paging does not necessarilyi mean your system is low on memory. The important metric to look at for low memory conditions is ’scan rate’, as this metric will tell you when the operating system is scanning memory pages to look for memory to page out.
File I/O paging is normal, as most modern operating systems will use available memory to cache files in order to prevent disk I/O and improve performance. In Solaris 10, the paging for file I/O can be observed by using the ‘-p’ option of vmstat. See page 138 in Solaris Performance and Tools
Paging is a problem when the memory used by your applications (called “anonymous memory” in Solaris) is getting paged to disk. Before this happens, however, you would see page scanner activity (sr, scan rate) in vmstat.
Virtual Memory (especially Java)
In java applications, memory bottlenecks cannot be detected using Operating System utilities such as vmstat, because the memory is managed by the process, or “virtual machine” (i.e. JVM - Java Virtual Machine). If your JVM is low on memory, then you will see “too much” time doing garbage collection. What is “too much?” Well, that depends on your application. A general rule-of-thumb is that your application should spend less than 5-10% of it time doing Garbage Collection.
To detect low-memory conditions in your Java application, you can spend a lot of money on fancy Java monitoring tools, or you can just enable verbose logging of garbage collection events, which has very little overhead. If your application is really low on memory, you may even see java.lang.OutOfMemory Exceptions being thrown (they should appear in the stderr log). Most times, however, you will only see decreasing performance, as the application spends more and more time collecting garbage.
Conclusion
Well, that is all for this (very) brief primer on detecting memory bottlenecks. By using vmstat and analyzing your java garbage collection, you should be able to detect most of your memory bottlenecks.
Good luck!
Additional Info
5 Responses to “Bottlenecks: Part 2 of 4”
I prefer to monitor JVM memory utilization by using the HEAP SIZE parameter (value). By monitoring free HEAP SIZE it’s easy to determine memory leaks in Java application. If free HEAP SIZE is decreased over time it’s a notification that something wrong with memory utilization.
By melnykenator on Nov 15, 2007
Thanks for the comment. Yes, the heap size can be monitored over time to identify the existence of a memory leak, but there is a difference between a memory leak and a memory bottleneck. An application could have a memory bottleneck (that is, a heap size that is too small) but not have a memory leak. An application with a memory leak, may not have a memory bottleneck (but if left to run long enough, it will eventually become a memory bottleneck, but it could take days, depending on how bad the leak is).
Either way, monitoring the verbose garbage collection data will provide more information than just heap size (before and after GC), you will also find the amount of time spent in GC, how many bytes were freed, the sizes of the different heap pools (e.g. Permament space, Tenured generation, Eden Space, etc.)
By cw on Nov 15, 2007
Hi Charlie,
You seem to discount (or ignore) that not all memory issues are related to leaks. In fact there is a large proportion related to capacity issues. Too many concurrent requests allocating and retaining significant amounts of memory during the execution life cycle of a request.
Also there is a big difference between knowing there is a memory issue and understanding why and when. OS specific tools rarely (more like never) allow sufficient correlation with application and system tasks. Context is important. Unless of course you merely what to classify an incident and not actual perform a problem analysis.
Recent Blog Entry on Memory Metrics
http://blog.jinspired.com/?p=33
Old Article
http://www.jinspired.com/products/jxinsight/outofmemoryexceptions.html
kind regards,
William Louth
JXInsight Product Architect
CTO, JINSPIRED
“Performance Management and Problem Diagnostics for Java EE, SOA, and Grid Computing”
http://www.jinspired.com
By william.louth on Nov 16, 2007
I meant to direct my reply to melnykenator.
“Performance Management and Problem Diagnostics for Java EE, SOA, and Grid Computing”
http://www.jinspired.com
By william.louth on Nov 16, 2007
You might be interested in the following blog entry I just published related to measuring object allocation costs which is important when estimating resource consumption for use cases within a software execution model constructed for the purpose of modeling performance and capacity.
How much does that Object cost in the VM?
http://blog.jinspired.com/?p=174
Kind regards,
William Louth
JXInsight Product Architect
CTO, JINSPIRED
“Performance Management and Problem Diagnostics for Java EE, SOA, and Grid Computing”
http://www.jinspired.com
By william.louth on Nov 18, 2007