PerformanceEngineer.com

For Sale

I’m finding that I no longer have the time to maintain this site to the degree that I would like. Is anyone interested in taking it off my hands? Tweet me a DM to twitter.com/perfeng

Posted in 27 | No Comments »

Performance is not a Check Box

This is a post I wrote for my company’s blog. Reproduced here in its entirety

For the IntraLinks Performance Engineering team, our ultimate goal is application performance that yields an excellent user experience. Such a broadly defined goal requires a broad approach. In order to effectively deliver an excellent user experience, performance must be managed throughout the application life-cycle. Taking a larger view of application performance opens up additional possibilities for improving user satisfaction that might be missed with a narrower focus. Application Performance is not just a testing activity, it is built-in to the product and therefore must be addressed throughout the product life-cycle in order to be effective and achieve the best results. Let’s look at some of the ways application performance is managed during the software development process.

Requirements

From the very beginning, the definition of requirements can have a tremendous impact on application performance. Traditionally falling into the general bucket of “non-functional requirements,” performance requirements are typically used as pass/fail criteria for later testing. For example:

• How many concurrent users must be supported?
• What is the expect throughput (transactions per second, bytes per second, etc)?
• What are the required response times?

Having these requirements defined up front is important to making sure the application architecture can support the defined specifications, and they give focus to performance testing efforts. However, taking a broader view of managing user experience, we must also consider the functional requirements and their potential impact. Even though they may not seem like performance requirements, there are many areas that must be evaluated. Here are just a few examples:

• What data sizes must be supported? Many performance issues are data-driven.
• What will be the supported browsers? Recent browsers perform much better than old.
• Can some complexity be simplified? Complexity increases risk of poor performance.
• Can asynchronous requests improve user experience? AJAX is an excellent example of improving user experience using asynchronous requests.
• How much data needs to be presented to the user? Can long lists of data be paginated?

Managing application requirements is the first step in managing performance.

Design and Architecture

Once the requirements are defined and agreed to, the performance engineers must understand the design and architecture of the solution, and identify potential risk areas. For example, a single-threaded process cannot be expected to satisfy hundreds of requests per second. Performance Engineers, Architects and Designers must carefully consider the design choices to ensure application responsiveness, scalability and stability:

• Use of frameworks, libraries, dependencies
• Understand potential architectural bottlenecks – database access, messaging, flow of data
• User Interface design

As is the case with all areas of software development, documentation and communication are key for performance engineers to gauge the impact of design and architecture on application performance. Armed with this knowledge, the performance engineer can more effectively identify root cause of performance issues, saving critical time in problem resolution.

Development

Performance defects are often hard to fix, making early detection a key factor in reducing product development and testing time. Some of the things we do to catch performance defects early include:

• Code analysis – PMD, FindBugs
• Integration Tests – Capture performance data during build integration testing
• Code profiling – Identify poor algorithms, slow SQL queries, etc.

Having performance engineers working closely and communicating with developers during this phase is critical not only for finding issues early, but also for creating effective performance tests. Comprehensive performance test plans are created during this phase, and having the implementation details is vital for identifying areas in the code that require deep testing.

Testing

By the time we get to the testing phase, after managing our application performance through requirements, design and implementation, we will have already identified and resolved many issues. There is much still to do, though, as scalability and concurrency issues can only be found in during large-scale testing in a suitable test environment, with production-like data. Furthermore, as code is changed, defects fixed, new requirements introduced, there is always a risk of introducing performance defects. Effective performance tests are critical to:

• Validate requirements are met
• Identify bottlenecks, concurrency issues
• Test scalability, stability

Equally as important as the actual test execution is the effective communication of the test results. Data + Context = Information, and performance testing activities generate massive amounts of data. Managing application performance requires having the proper context (requirements, design, architecture, etc) to make the performance data useful information.

Production

Application performance management doesn’t stop after testing has been completed and the application is deployed and live in production. Once the application is live, the user experience must be measured and monitored in order to identify any performance problems that were not previously caught. Additionally, monitoring in production is absolutely critical for ongoing application performance management in order to validate the accuracy of performance tests, scalability predictions and usage models. Live production data provides key feedback to help refine and improve all of the previous performance management activities – requirements, design, architecture and testing. It is impossible to perfectly replicate or predict exactly how users will access the application and there will always be some issues that arise in production, so it is important to have the ability to detect performance issues as they happen and collect actionable data so the issue can be resolved quickly and at minimal cost.

More than Just Testing

Managing Application Performance is more than just performance testing, it is managing the user experience. It requires a comprehensive approach and involvement throughout the product life cycle. Performance is not a line item in a list of to-dos. Performance is not a check box.

Posted in 27 | 4 Comments »

Performance Testing from the Cloud

I’ve seen a lot of activity lately about using “the cloud” for performance testing. Here’s the current run-down of load testing options “from the cloud:”

Open Source

• Pylot – Web Load Testing from Amazon Elastic Compute Cloud (EC2)
• Grinder In The Cloud – A cloud based load testing environment
• Load Testing From The Cloud (JMeter)

Pay to play

• SOASTA
• BrowserMob
• KeyNote
• LoadStorm

It’s exciting to see so many new options cropping up. I would not be surprised to see some of the other smaller players in the performance testing space start offering similar services this year, while the larger (HP, Borland) companies will be unable to move quickly enough away from their existing “consulting services.” The open source options appear to be well behind the current commercial offerings, but could be viable options if you’re willing to put in the time.

___________
UPDATE: 2/24/09 2:08 PM — Added LoadStorm
___________
UPDATE: 3/13/09 10:14 AM — Corrected URL for “Grinder In The Cloud …”

Posted in Test Tools | 5 Comments »

The Art of Application Performance Testing

Book Description

This practical book provides a step-by-step approach to testing mission-critical applications for scalability and performance before they’re deployed — a vital topic to which other books devote one chapter, if that.

Businesses today live and die by network applications and web services. Because of the increasing complexity of these programs, and the pressure to deploy them quickly, many professionals don’t take the time to ensure that they’ll perform well and scale effectively. The Art of Application Performance Testing explains the complete life cycle of the testing process, and demonstrates best practices to help you plan, gain approval for, coordinate, and conduct performance tests on your applications. With this book, you’ll learn to:

* Set realistic performance testing goals
* Implement an effective application performance testing strategy
* Interpret performance test results
* Cope with different application technologies and architectures
* Use automated performance testing tools
* Test traditional local applications, web-based applications, and web services (SOAs)
* Recognize and resolves issues that are often overlooked in performance tests

Written by a consultant with 30 years of experience in the IT industry and over 12 years experience with performance testing, this easy-to-read book is illustrated with real-world examples and packed with practical advice. The Art of Application Performance Testing thoroughly explains the pitfalls of an inadequate testing strategy and offers you a robust, structured approach for ensuring that your applications perform well and scale effectively when the need arises.

Posted in Books | No Comments »

The Art of Capacity Planning: Scaling Web Resources

Book Description

Success on the web is measured by usage and growth. Web-based companies live or die by the ability to scale their infrastructure to accommodate increasing demand. This book is a hands-on and practical guide to planning for such growth, with many techniques and considerations to help you plan, deploy, and manage web application infrastructure. The Art of Capacity Planning is written by the manager of data operations for the world-famous photo-sharing site Flickr.com, now owned by Yahoo! John Allspaw combines personal anecdotes from many phases of Flickr’s growth with insights from his colleagues in many other industries to give you solid guidelines for measuring your growth, predicting trends, and making cost-effective preparations. Topics include:

• Evaluating tools for measurement and deployment
• Capacity analysis and prediction for storage, database, and application servers
• Designing architectures to easily add and measure capacity
• Handling sudden spikes Predicting exponential and explosive growth
• How cloud services such as EC2 can fit into a capacity strategy

In this book, Allspaw draws on years of valuable experience, starting from the days when Flickr was relatively small and had to deal with the typical growth pains and cost/performance trade-offs of a typical company with a Web presence. The advice he offers in The Art of Capacity Planning will not only help you prepare for explosive growth, it will save you tons of grief.

Posted in Books | No Comments »

Performance Engineering Podcast

Software Engineering Radio has a podcast on Performance Engineering.

In this episode Martin talks with Chris Grindstaff about the fundamentals of performance engineering. The episode discusses when and how to work on performance of client- and server-side systems, what you should take into account during development to avoid performance issues, typical situations that cause performance problems, and some common pitfalls when analysing performance.

It’s worth a listen

Posted in General, Performance | No Comments »

New Site Feature: Careers

I’ve added a new feature to the site: Careers

If you’re in the market, keep an eye on this page and I’ll post job openings as I receive them. If you would like to list a job opening, please send an email to job-listings@performanceengineer.com

Posted in Careers | 1 Comment »

Orbitz Open Sources Monitoring Tools ERMA and Graphite

I saw this recent article about some new tools from Orbitz: InfoQ: Orbitz Open Sources Monitoring Tools ERMA and Graphite

Graphite looks particularly interesting as it has features overcoming some of the things I don’t like about RRDtool:

Graphite is a Python web application that has been developed to provide scalable storage and visualization for numeric time-series data. It receives the output from the CEP engine, which consists of data aggregated for over 70,000 metrics. A Graphite portlet was developed in order to integrate with a monitoring portal, which is based on JBoss Portal framework. The portal presents tabular and graphical views of vital system statistics and RSS feeds for alarms. Users can subscribe to feeds for particular alarm severities and/or affected applications. The Graphite web application itself is used to display visual graphs of monitoring statistics generated using RESTful URLs. Graphite data can be presented as line graphs, pie charts or raw CSV data. Graphite also provides a web-based command line interface, which power users can use to very quickly and easily create and share dashboards containing collections of related graphs.

I’ll hopefully find some time to try this out

Posted in Monitoring, tools | 2 Comments »

Extract Transaction Data From LoadRunner Analysis

I had a need to extract transaction response time data from a bunch of LoadRunner Analysis files, and I really didn’t want to do endless cut and paste operations from within the LoadRunner Analysis tool. I created this Python script to extract transaction response time data from the LoadRunner Analysis mdb file and output into CSV format, which can then be imported into MySQL, Excel, etc. The analysis file is a MS Access JET Database which allows the use of Python’s win32com module to access the data.

# lr_extract.py
# --------------------------------------------------------------------
# Extracts Transaction response time data from the MS Jet database file (.mdb)
# created by LoadRunner Analysis tool
#
# Requires DAO 3.6 library.
# --------------------------------------------------------------------
# Usage: python lr_extract.py lr_analysis.mdb
import sys
import string
import pythoncom
import win32com.client

const = win32com.client.constants

daoEngine = win32com.client.Dispatch('DAO.DBEngine.36')
db = daoEngine.OpenDatabase(sys.argv[1])

query = """
select
    [Result].[Start Time]+[Event_meter].[End Time] as t_stamp,
    [Event_map].[Event Name] as txn_name,
    [Event_meter].[Value] as resp_time,
    [Result].[Result Name] as result_name
from
    Result,
    Event_map,
    Event_meter
where
    [Event_map].[Event Type] = 'Transaction' and
    [Event_meter].[Event ID] = [Event_map].[Event ID]
order by [Result].[Start Time]+[Event_meter].[End Time]
"""
rs = db.OpenRecordset(query)

print("timestamp,transaction,response time,result name")
while not rs.EOF:
    print("%s,%s,%s,%s" % (rs.Fields[0],rs.Fields[1],rs.Fields[2],rs.Fields[3]) )
    rs.MoveNext()

Posted in Python, Tips & Tricks, loadrunner | 2 Comments »

Introduction to The Grinder

The Grinder is:

The Grinder is a Java^TM load testing framework that makes it easy to run a distributed test using many load injector machines.

Until recently, I had been reluctant to investigate The Grinder because I was unfamiliar with Jython/Python, and hesitant to dig into a new language. I was wrong.

Not only am I happy to have discovered that Python and Jython are not too difficult to pick up, but The Grinder is quite a flexible tool that has a lot of potential and I would say a nearly capable replacement for LoadRunner from a load generation perspective.

In this article I’ll review how to setup The Grinder and use Eclipse to edit your scripts. 

Read the rest of this entry »

Posted in Test Tools | 13 Comments »