Performance Testing
Performance testing is one of those phrases on which there is little
agreement as to what it precisely means. However the various
descriptions given for performance testing show that there is a broad
agreement about the non-functional requirements that need to be tested.
Another complication factor is that the various types of performance
testing are all done using a load testing tool. The tool, like all
tools, enables somebody to do a job, not to do the testing itself. For
this you need a trained and experienced performance or load tester.
Part-1
Performance Testing and Non-Functional Requirements
The performance based non-functional requirements to be tested cover a
number of questions which the owners of the system can reasonable ask.
These include:
- How many users can the system deal with?
- What performance will each type of transaction have?
- What happens if very high usage occurs?
- What happens if the system fails?
- Can the system operate reliably over a period of time?
Each of these questions then has detail numbers placed against them in order to allow testing to take place.
Types of Performance Testing
For each of the questions a separate type of test can be defined. The
most important confusion is between load and performance testing. Some
documentation does not mention one or the other, some swap meanings
which results in a general lack of clarity. In essence the two types of
testing are mirror images of each other.
This article argues that although a load testing tool is used for
each type of testing, the key difference between them is the Acceptance
Criteria or objectives of each type of test. The types of test matching
the non-functional questions are:
- Load Testing – How many users supported.
- Performance Testing – The performance of each transaction type.
- Stress Testing – Explores high but valid usage.
- Robustness Testing – Explores the effect of unavailability of resources.
- Endurance Testing – Checks long term running of the system.
To illustrate each type of testing a simple real world model will be
used. Imagine our system is a road network connecting together houses
with shops, schools and workplaces. Our transactions are users going
from one point in the network to another, such as from the house to the
workplace. More complicated transactions might be leaving the house to
go to the school to drop off children and from there drive on to work.
The functional testing that would have been done would checked that each
type of transaction is possible, that roads exist between each of the
network points for each type of transactions they want to follow. The
following paragraphs will illustrate how the various types of load
testing would apply to this model.
Load Testing
Load Testing means that given a performance range can a varying
numbers of users be dealt with. This is dealing with the issues of
checking various patterns of load and seeing if a pattern of usage is
delivered rather than checking the details of the individual
transactions.
In terms of the road network model this would mean simulating a wide
variety of patterns of usage. Can the school and work rush peak load
perform acceptably. Can picking up children during school homeward rush
be dealt with. Can delivery vans get to shops in time and not block the
roads. These are all issues to do with checking the volume and mix of
transactions.
Performance Testing
Performance Testing itself checks that for a given range of users
loads are each of the specific transaction performance criteria met.
This is involved in seeing where the problems are with individual
transactions, rather than the delivery of the overall load.
The road network would show this by tracking individual users during
the various road patterns and see if they get to school or work on time.
Whether they are held up in traffic jams, and where the problems occur
so that changes can be made to the road network.
Stress Testing
Stress Testing explores the issues of what happens to the system as
it is put under valid load in excess of it design parameters. It is
sometimes defined as to break the system but this is not the point of
the test, though it may be the result. The purpose is to put the system
under a number of very high load patterns and observe what happens to
it. Some of the observations would be if the system just slows down,
come to a stop, or causes serious damage.
For a road network this would be the equivalent of putting in a lot
more vehicles. Perhaps the school is having an exchange visit, there is a
conference in town, or the shops are having a special sale which
attracts out of town visitors. Each or several of these would put a
stress on the network.
Robustness Testing
Robustness Testing in contrast to Stress Testing deals with the
response of the system under various loads when resources, such as a
dependent system, hardware, or network, become unavailable. Again the
purpose is not to break the system but study the system’s response to
the various resource failures against several load patterns.
The equivalent for the road network would be having roads collapse,
schools or works places shut down, or the shops deciding not the open.
The purpose is to see if everybody could still get around town.
Endurance Testing
Endurance Testing checks how stable the system is over a period of
time. It applies a high but normal load pattern of transactions to the
system for a continuous period. Checks are made during the run to see if
any subtle errors build up over a period of time which can affect the
performance or integrity of the system.
In road network terms this would be checking usage over several
consecutive days to see what happens. It may be that vehicles break down
and stop people getting around, or because of extended jams schools and
shops do not open on time. These errors build up over a period of time.
Load Testing Tools
All these tests use a load testing tool to assist in running of the
tests. The Acceptance Criteria and tests have to be designed by the
testers, and then implemented in the tool. The tool itself has three key
functions:
- Provide a simulated load to the system which can vary with time.
- Record the results of running test loads.
- Provide tools to display the results of tests.
In road terms they would be providing more vehicles to go on runs
within the network, having watchers to count vehicles and record what is
going on, and then writing a report on what is found.
When to Performance Test
- Types of Testing employed during a project including when performance testing takes place.
- User Requirements - three areas to include, such as non-functional requirements, and one to avoid when writing user requirements.
Part-2
Before starting the Performance testing,
performance test engineer should collect non-functional requirement
and Below is comprehensive list of performance Testing Requirements.
- Total user base : How many users will have access to this application?
- Customer will provide answer, how many users are having an access to the application.
- If the application is existing one, Customer will get the data from previous years to understand how many users are using the application. There are third party tools like Web trends to analyze the traffic of existing used application
- If the application is new and not yet in live, Business analyst and Application owners will be able to provide total user base.
- Concurrent User Load :What is the number of total users who will access the application together at any given point in time?
- According to industry standard, it is 10% of total users will be the concurrent load but this formula will not applicable for all the domains.
- Third party tools like web trends will be useful
- Business Processes ( Performance Scenarios):
- What are the business processes (Workflows) to be performance tested with concurrent users? Objective here is to find few business processes which can cover up to 80% of the application code"
- Load Distribution :What is the user/Load distribution on each business processes listed
- What is the time duration taken for each business process to complete end to end(Workflow)?
- How often are the users doing each business process (Workflow) in an hour, a day, etc? to get this data for existing application
- Service Level Agreement (Performance - Service Level Agreement):
- What are the acceptable response times for each transaction (page) in all the business flows that needs to be performance tested? Mention SLA for against each of the transaction (page) identified
- 1. All pages within the portal shall load in 3 seconds or less.
- 2. Report should load in 5 sec
- 3. Yearly Report should load in 20
- 4. SecgetAccountHistory API’s roundtrip response time should be < 250 msec
