PerformanceEngineer.com

Bottlenecks: Part 1 of 4

This is the first in a four part series about finding bottlenecks in an application or system.

Finding Bottlenecks

Finding bottlenecks in a system is one of the primary tasks that a performance engineer or tester performs and, I would argue, one of the most important. Yet is seems that few know the basics of bottlenecks and how to idenity them. Did you know that there are really only 4 different areas where you find nearly all bottlenecks?

Definition

What is a bottleneck? To put it as simply as I can, a bottleneck is an area of a system that limits the system's capacity or ability to meet performance expectation as load is increased.

Bottleneck Categories

When trying to identify a system bottleneck, it is important to keep in mind that the most common bottlenecks will in one of only four possible areas. This is important, because it will help you to systematically analyze and interpret the data you collect and make an accurate determination. Keep in mind that each tier in your system (web, application, database, etc.) is susceptible to any of the bottlenecks, and it is critical that if you are observing a bottleneck, you account for each tier. So, what are the bottleneck areas?

• CPU
• Memory
• I/O
• Software

Let's look at each one individually.

CPU Bottleneck

This is the obvious one and the one that everyone thinks they know. The CPU bottleneck is probably the most common (although I have no actual data to prove this hypothesis) and also probably the easiest to detect. In fact, I would argue that having a CPU bottleneck should be a design goal, because in that case, increasing capacity is as simple as adding CPUs or upgrading to faster CPUs.

To detect a CPU bottleneck, there are numerous tools available. On UNIX and Linux, some of the more common utilities are vmstat, sar, iostat. On Windows, perfmon.exe is typically used. A general rule of thumb to determine whether you have a CPU bottleneck is if your total CPU utilization is over 90% for single processor systems, and about 80% for SMP systems. Above these levels, the CPUs are so busy that threads begin queuing and performance suffers.

[Added 4/20/07: See also comments below for additional CPU bottleneck]

Fixing the CPU bottleneck

In order to remove the CPU bottleneck, additional CPU processing power must be added to the system, or the software running on the system needs to be optimized to use fewer CPU cycles. Typically, the easiest way is to add CPUs (either horizontally - additional server - or vertically - additional CPUs in the same server) or to upgrade to faster CPUs.

Next in the series

Next in the series…Memory bottlenecks