Performance Testing and Non-Functional Requirements

Performance Testing

Performance testing is one of those phrases on which there is little agreement as to what it precisely means. However the various descriptions given for performance testing show that there is a broad agreement about the non-functional requirements that need to be tested.

Another complication factor is that the various types of performance testing are all done using a load testing tool. The tool, like all tools, enables somebody to do a job, not to do the testing itself. For this you need a trained and experienced performance or load tester.

 

Part-1

Performance Testing and Non-Functional Requirements

The performance based non-functional requirements to be tested cover a number of questions which the owners of the system can reasonable ask. These include:

• How many users can the system deal with?
• What performance will each type of transaction have?
• What happens if very high usage occurs?
• What happens if the system fails?
• Can the system operate reliably over a period of time?

Each of these questions then has detail numbers placed against them in order to allow testing to take place.

Types of Performance Testing

For each of the questions a separate type of test can be defined. The most important confusion is between load and performance testing. Some documentation does not mention one or the other, some swap meanings which results in a general lack of clarity. In essence the two types of testing are mirror images of each other.

This article argues that although a load testing tool is used for each type of testing, the key difference between them is the Acceptance Criteria or objectives of each type of test. The types of test matching the non-functional questions are:

• Load Testing – How many users supported.
• Performance Testing – The performance of each transaction type.
• Stress Testing – Explores high but valid usage.
• Robustness Testing – Explores the effect of unavailability of resources.
• Endurance Testing – Checks long term running of the system.

To illustrate each type of testing a simple real world model will be used. Imagine our system is a road network connecting together houses with shops, schools and workplaces. Our transactions are users going from one point in the network to another, such as from the house to the workplace. More complicated transactions might be leaving the house to go to the school to drop off children and from there drive on to work. The functional testing that would have been done would checked that each type of transaction is possible, that roads exist between each of the network points for each type of transactions they want to follow. The following paragraphs will illustrate how the various types of load testing would apply to this model.

Load Testing

Load Testing means that given a performance range can a varying numbers of users be dealt with. This is dealing with the issues of checking various patterns of load and seeing if a pattern of usage is delivered rather than checking the details of the individual transactions.

In terms of the road network model this would mean simulating a wide variety of patterns of usage. Can the school and work rush peak load perform acceptably. Can picking up children during school homeward rush be dealt with. Can delivery vans get to shops in time and not block the roads. These are all issues to do with checking the volume and mix of transactions.

Performance Testing

Performance Testing itself checks that for a given range of users loads are each of the specific transaction performance criteria met. This is involved in seeing where the problems are with individual transactions, rather than the delivery of the overall load.

The road network would show this by tracking individual users during the various road patterns and see if they get to school or work on time. Whether they are held up in traffic jams, and where the problems occur so that changes can be made to the road network.

Stress Testing

Stress Testing explores the issues of what happens to the system as it is put under valid load in excess of it design parameters. It is sometimes defined as to break the system but this is not the point of the test, though it may be the result. The purpose is to put the system under a number of very high load patterns and observe what happens to it. Some of the observations would be if the system just slows down, come to a stop, or causes serious damage.

For a road network this would be the equivalent of putting in a lot more vehicles. Perhaps the school is having an exchange visit, there is a conference in town, or the shops are having a special sale which attracts out of town visitors. Each or several of these would put a stress on the network.

Robustness Testing

Robustness Testing in contrast to Stress Testing deals with the response of the system under various loads when resources, such as a dependent system, hardware, or network, become unavailable. Again the purpose is not to break the system but study the system’s response to the various resource failures against several load patterns.

The equivalent for the road network would be having roads collapse, schools or works places shut down, or the shops deciding not the open. The purpose is to see if everybody could still get around town.

Endurance Testing

Endurance Testing checks how stable the system is over a period of time. It applies a high but normal load pattern of transactions to the system for a continuous period. Checks are made during the run to see if any subtle errors build up over a period of time which can affect the performance or integrity of the system.

In road network terms this would be checking usage over several consecutive days to see what happens. It may be that vehicles break down and stop people getting around, or because of extended jams schools and shops do not open on time. These errors build up over a period of time.

Load Testing Tools

All these tests use a load testing tool to assist in running of the tests. The Acceptance Criteria and tests have to be designed by the testers, and then implemented in the tool. The tool itself has three key functions:

1. Provide a simulated load to the system which can vary with time.
2. Record the results of running test loads.
3. Provide tools to display the results of tests.

In road terms they would be providing more vehicles to go on runs within the network, having watchers to count vehicles and record what is going on, and then writing a report on what is found.

When to Performance Test

• Types of Testing employed during a project including when performance testing takes place.
• User Requirements - three areas to include, such as non-functional requirements, and one to avoid when writing user requirements.

 

Part-2

Before starting the Performance testing, performance test engineer should collect non-functional requirement and Below is comprehensive list of performance Testing Requirements.

•  Total user base : How many users will have access to this application? 
• Customer will provide answer, how many users are having an access to the application.
• If the application is existing one, Customer will get the data from previous years to understand how many users are using the application. There are third party tools like Web trends to analyze the traffic of existing used application
• If the application is new and not yet in live, Business analyst and Application owners will be able to provide total user base.
• Concurrent User Load :What is the number of total users who will access the application together at any given point in time?
• According to industry standard, it is 10% of total users will be the concurrent load but this formula will not applicable for all the domains.
• Third party tools like web trends will be useful
• Business Processes ( Performance Scenarios):  
• What are the business processes (Workflows) to be performance tested with concurrent users? Objective here is to find few business processes which can cover up to 80% of the application code"
• 
  
• Load Distribution :What is the user/Load distribution on each business processes  listed 
• What is the time duration taken for each business process to complete end to end(Workflow)? 
• How often are the users doing each business process (Workflow) in an hour, a day, etc? to get this data for existing application
• Service Level Agreement (Performance - Service Level Agreement):
• What are the acceptable response times for each transaction (page) in all the business flows that needs to be performance tested? Mention SLA for against each of the transaction (page) identified
• 1. All pages within the portal shall load in 3 seconds or less.
• 2. Report should load in 5 sec
• 3. Yearly Report should load in 20 
• 4. SecgetAccountHistory API’s roundtrip response time should be < 250 msec