SAP Performance/Volume testing

Its is always recommended to performance/volume test your SAP applications as SAP systems has specifically configured to the critical business processes. A successful performance testing confirms that underlying hardware and infrastructure is able to support the required transaction volume and able to generate desired response time for the critical processes.

SAP provides a wide range of products:

• Customer Relationship Management (CRM)
• Enterprise Resource Planning (ERP)
• Product Lifecycle Management (PLM)
• Supply Chain Management (SCM)
• Supplier Relationship Management (SRM)

SAP's products focus on Enterprise Resource Planning (ERP). The company has no main product as different products serve to meet the different needs of businesses. Now a days businesses are looking for a software product that handle all the companies needs, from inventory management, invoicing customers, sales and distribution and controlling the whole business planning in one module. This need from businesses lead to the development of SAP Enterprise Central Component [ECC] system. SAP ECC offers the opportunity to integrate all software data programs by all departments by using one single database.

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

Following diagram depicts the 3-tier architecture of a typical SAP system

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

Effective performance test planning plays a vital role in the success of a SAP performance/volume test project. Outcome of the performance test planning phase should indicate the things and order of activities need to be performed. Typical outputs of a planning phase are performance test strategy/ test plan, critical business scenarios, Targeted transaction volume and response time, SLAs, Test data requirements, Risks and mitigation plan etc.

Non functional requirement (NFR) documents is one of the key documents which may provide next level of details while planning, NFR should state all performance/volume testing related requirements very specifically like Expected transaction response time or Dialog response time or work process response time. Different architects may use different terminologies while referring to response time in the world of SAP projects, So it is recommended to understand the requirement very specifically.

 

Effective and realistic performance/volume testing requires to test the SAP systems with two main processes which is Online users and Batches/reports. Batches plays an important role while we are doing performance test to identify the capabilities of underlying hardware and infrastructure. Its is recommended to understand the 24Hrs round the clock load profile of the system and to identify the points where there is overlap between peak online user load and batches running in parallel. Along with this standalone batch jobs performance testing is also critical for the daily/weekly/monthly and yearly batches which are business critical or resource intensive. NFR document should explicitly mention the expectation and SLAs for the batches.

Based on the identifies business scenarios/batches, performance test could be conducted with three possible combinations

1. Standalone peak online user load test.

2. Peak Online+ Batches performance test

3. Standalone batch performance test

 

Batches and the interfaces with the legacy/External and third party systems also plays a key role in the performance of the SAP system, we should identify the times where we may received significant load coming from these systems and SAP system has to process the scheduled and ad-hoc requests arriving from these systems.

Successful testing of batches will give us indication of the records or data processed within a given period of time, batches are more prone to have performance deficiency and highly vulnerable to the poor SQL queries, work process consumption, high number of enque and lock entries etc.

 

Another important factor in planning the performance and volume test in test data, You should have a clear understanding of the load profile you are going to simulate as part of performance testing. Its is must recommended that test environment should have the data similar to production and volume test results will impact a lot if these are performance with lesser data in test environments compared to production. For newly built system you may have zero or very few records in the database which is expected to grow over the time, in such cases there will be a difference in the response time in production and the one you received during performance testing when testing was conducted on empty or with very few records on database.

Expectation should be set clearly with the stakeholders and project management like who is going to create the data and how data will be populated into the test environment and the efforts required to create test data.

There also should be a clear understanding of who is going to create the user IDs with appropriate role and number of user IDs required. User IDs with improper role may not be able to perform the required operation. Role based testing should also be discussed with stakeholders very specifically.

Following diagrams depicts a typical performance test lab setup

 

 

Please note that load generators running SAP GUI users should have SAP GUI installed on the machines. System resource requirement to run a single SAP GUI user is much higher than a typical web user, based on the specifications of your load generators machines you should calculate the footprint of the system resource utilization by a single SAP GUI user and accordingly you van calculate the Load generator machines required to accommodate your needs.

 

Mitigation of Performance Testing Impediments

An impediment is anything that prevents people from doing their job. Here are some impediments that performance testing teams have encountered.

A. Unavailability of subject matter / technical experts such as developers and operations staff.

B. Unavailability of applications to test due to delays or defects in the functionality of the system under test.

C. Lack of Connectivity/access to resources due to network security ports being available or other network blockage.

D. The script recorder fails to recognize applications (due to non-standard security apparatus or other complexity in the application).

E. Not enough Test Data to cover unique conditions necessary during runs that usually go several hours.

F. Delays in obtaining or having enough software licenses and hardware in the performance environment testing.

G. Lack of correspondence between versions of applications in performance versus in active development.

H. Managers not familiar with the implications of ad-hoc approaches to performance testing.

Fight Or Flight? Proactive or Reactive?

      Some call the list above "issues" which an organization may theoretically face.

      Issues become "risks" when they already impact a project.

      A proactive management style at a particular organization sees value in investing up-front to ensure that desired outcomes occur rather than "fight fires" which occur without preparation.

      A reactive management style at a particular organization believes in "conserving resources" by not dedicating resources to situations that may never occur, and addressing risks when they become actual reality.

  
Subject Matter Expertise

      The Impediment
      Knowledge about a system and how it works are usually not readily available to those outside the development team.

      What documents written are often one or more versions behind what is under development.

      Requirements and definitions are necessary to separate whether a particular behavior is intended or is a deviation from that requirement.

      Even if load testers have access to up-to-the-minute wiki entries, load testers usually are not free to interact as a peer of developers.

      Load testers are usually not considered a part of the development team or even the development process, so are therefore perceived as an intrusion to developers.

      To many developers, Performance testers are a nuisence who waste time poking around a system that is "already perfect" or "one we already know that is slow".

      What can reactive load testers do?
      Work among developers and easedrop on their conversations (like those studying animals in the wild).

      What can proactive load testers do?
      Up-front, an executive formally establishes expectations for communication and coordination between developers and load testers.

      Ideally, load testers participate in the development process from the moment a development team is formed so that they are socially bonded with the developers.

      Recognizing that developers are under tight deadlines, the load test team member defines exactly what is needed from the developer and when it is needed.

      This requires up-front analysis of the development organization:

          o the components of the application
          o which developers work on which component
          o contact information for each developer
          o existing documents available and who wrote each document
          o comments in blogs written by each developer

      An executive assigns a "point person" within the development organization who can provide this information.

      Assignments for each developer needs to originate from the development manager under whom a developer works for.

            When one asks/demands something without the authority to do so, that person would over time be perceived as a nuisence.

            No one can serve two masters. For you will hate one and love the other; you will be devoted to one and despise the other.

      A business analyst who is familiar with the application's intended behavior makes a video recording of the application using a utility such as Camtasia from ToolSmith. A recording has the advtange of capturing the timing as well as the steps.

               

The U.S. military developed the web-based CAVNET system to collaborate on innovations to improvise around impediments found in the found.


Availability of applications

      The Impediment
      Parts of an applications under active development become inacessible while developers are in the middle of working on them.

      The application may not have been built successfully. There are many root causes for bad builds:

          o Specification of what goes into each build are not accurate or complete.
          o Resources intended to go into a particular build are not made available.
          o An incorrect version of a component is built with newer incompatible components.
          o Build scripts and processes do not recognize these potential errors, leading to build errors.
          o Inadequate verification of build completeness.

      What can reactive load testers do?
      Frequent (nightly) builds may enable testers more opportunities than losing perhaps weeks wait for the next good build.

      Testers switch to another project/application when one application cannot be tested.

      What can proactive load testers do?
      Use a separate test environment that is updated from the development system only when parts of the application become stable enough to test.

      Have a separate test environment for each version so that work on a prior version can occur when a build is not successful on one particular environment.

      Develop a "smoke test" suite to ensure that applications are testable.

      Coordinate testing schedules with what is being changed by developers.

      Analyze the root causes of why builds are not successful, and track progress on elminating those causes over time.

               
Connectivity/access to resources

      The Impediment
      Workers may not be able to reach the application because of network (remote VPN) connectivity or security access.

      What can reactive load testers do?
      Work near the physical machine.

      Grant unrestricted access to those working on the system.

      What can proactive load testers do?
      Analyze the access for each functionality required by each role.

      Pre-schedule when those who grant access are available to the project.

               
 Script Recorder Recognition

      The Impediment
      Load test script creation software such as LoadRunner work by listening and capturing what goes across the wire and display those conversations as script code which may be modified by humans.

      Such recording mechanisms are designed to recognize only standard protocols going through the wire.

      Standard recording mechanisms will not recognize custom communications, especially within applications using advanced security mechanisms.

      Standard recording mechanisms also have difficulty recognizing complex use of Javascript or CSS syntax in SAP portal code.

      What can reactive load testers do?
      Skip (de-scope) portions which cannot be easily recognized.

      What can proactive load testers do?
      To ensure that utility applications (such as LoadRunner) can be installed, install them before locking down the system.

      Define the pattern install them before locking down the system.

               
 Test Data

      The Impediment
      Applications often only allow a certain combination of values to be accepted. An example of this is only specific postal zip codes being valid within a certain US state.

      Using the same value repeatedly during load testing does not create a realistic emulation of actual behavior because most modern systems cache data in memory, which is 100 times faster than retrieving data from a hard drive.

      This discussion also includes role permissions having a different impact on the system. For example, the screen of an administrator or manager would have more options. The more options, the more resources it takes just to display the screen as well as to edit input fields.

      A wide variation in data values forces databases to take time to scan through files. Specifying an index used to retrieve data is the most common approach to make applications more efficient.

      Growth in the data volume handled by a system can render indexing schemes inefficient at the new level of data.

      What can reactive load testers do?
      Use a single role for all testing.

      Qualify results from each test with the amount of data used to conduct each test.

      Use trial-and-error approachs to finding combinations of values which meet field validation rules.

      Examine application source code to determine the rules.

      Analyze existing logs to define the distribution of function invocations during test runs.

      What can proactive load testers do?
      Project the likely growth of the application in terms of impact to the number of rows in each key data variable. This information is then used to define the growth in row in each table.

      Define procedures for growing the database size, using randomized data values in names.

               
 Test Environment

      The Impediment
      Creating a separate enviornment for load testing can be expensive for a large complex system.

      In order to avoid overloading the production network, the load testing enviornment is often setup so no communication is possible to the rest of the network. This makes it difficult to deploy resources into the environment and then retrieve run result files from the environment.

      A closed environment requires its own set of utility services such as DNS, authentication (LDAP), time sychronization, etc.

      What can reactive load testers do?
      Change network firewalls temporarily while using the development environment for load testing (when developers do not use it).

      Use the production fail-over environment temporarily and hope that it is not needed during the test.

      What can proactive load testers do?
      Build up a production environment and use it for load testing before it is used in actual production.

               
Correspondance Between Versions

      The Impediment
      Defects found in the version running on the perftest environment may not be reproducible by developers in the development/unit test environments running a different (more recent) version.

      Developers may have moved on to a different version, different projects, or even different employers.

      What can reactive load testers do?
      Rerun short load tests on development servers. If the server is shared, the productivity of developers would be affected.

      What can proactive load testers do?
      Before testing, freeze the total state of the application in a full back-up so that the exact state of the system can be restored, even after changes are tried to diagnose or fix the application on the system where it's found.

      Run load tests with trace logs information. This would not duplicate how the system is actually run in production mode.

               
 Ad-hoc Approaches

      The Impediment
      Most established professional fields (such as accounting and medicine) have laws, regulations, and defined industry practices which give legitimacy to certain approaches. People are trained to follow them. The consequences of certain courses of action are known.

      But the profession of performance and load testing has not matured to that point.

      The closest industry document, ITIL, is not yet universally adopted. And ITIL does not clarify the work of performance testing in much detail.

      Consequently, each individual involved with load testing is likely to have his/her own opinions about what actions should be taken.

      This makes rational exploration of the implications of specific courses of action a conflict-ridden and thus time-consuming and expensive endeavor.

      What can reactive load testers do?
      Allocate time for planning before starting actual work until concurrance on the project plan is achieved among the stakeholders.

      Revise project completion estimates or scope as new information becomes available.

      What can proactive load testers do?
      Before the project gets away, agree on the rationale for elements of the project plan and who will do what when (commitments of tasks and deliverables). This is difficult for those who are not accustomed to being accountable, and requests for it would result in withdrawl or other defensive behavior.

      Identify alternative approaches and analyze them before managers come up with it themselves.

      Up-front, identify how to contact each stakeholder and keep them updated at least weekly, and immediately if decisions impact what they are actively working on.

      If a new manager is inserted in the project after it starts, review the project plan and rationale for its elements.

Performance Testing and Non-Functional Requirements

Performance Testing

Performance testing is one of those phrases on which there is little agreement as to what it precisely means. However the various descriptions given for performance testing show that there is a broad agreement about the non-functional requirements that need to be tested.

Another complication factor is that the various types of performance testing are all done using a load testing tool. The tool, like all tools, enables somebody to do a job, not to do the testing itself. For this you need a trained and experienced performance or load tester.

 

Part-1

Performance Testing and Non-Functional Requirements

The performance based non-functional requirements to be tested cover a number of questions which the owners of the system can reasonable ask. These include:

• How many users can the system deal with?
• What performance will each type of transaction have?
• What happens if very high usage occurs?
• What happens if the system fails?
• Can the system operate reliably over a period of time?

Each of these questions then has detail numbers placed against them in order to allow testing to take place.

Types of Performance Testing

For each of the questions a separate type of test can be defined. The most important confusion is between load and performance testing. Some documentation does not mention one or the other, some swap meanings which results in a general lack of clarity. In essence the two types of testing are mirror images of each other.

This article argues that although a load testing tool is used for each type of testing, the key difference between them is the Acceptance Criteria or objectives of each type of test. The types of test matching the non-functional questions are:

• Load Testing – How many users supported.
• Performance Testing – The performance of each transaction type.
• Stress Testing – Explores high but valid usage.
• Robustness Testing – Explores the effect of unavailability of resources.
• Endurance Testing – Checks long term running of the system.

To illustrate each type of testing a simple real world model will be used. Imagine our system is a road network connecting together houses with shops, schools and workplaces. Our transactions are users going from one point in the network to another, such as from the house to the workplace. More complicated transactions might be leaving the house to go to the school to drop off children and from there drive on to work. The functional testing that would have been done would checked that each type of transaction is possible, that roads exist between each of the network points for each type of transactions they want to follow. The following paragraphs will illustrate how the various types of load testing would apply to this model.

Load Testing

Load Testing means that given a performance range can a varying numbers of users be dealt with. This is dealing with the issues of checking various patterns of load and seeing if a pattern of usage is delivered rather than checking the details of the individual transactions.

In terms of the road network model this would mean simulating a wide variety of patterns of usage. Can the school and work rush peak load perform acceptably. Can picking up children during school homeward rush be dealt with. Can delivery vans get to shops in time and not block the roads. These are all issues to do with checking the volume and mix of transactions.

Performance Testing

Performance Testing itself checks that for a given range of users loads are each of the specific transaction performance criteria met. This is involved in seeing where the problems are with individual transactions, rather than the delivery of the overall load.

The road network would show this by tracking individual users during the various road patterns and see if they get to school or work on time. Whether they are held up in traffic jams, and where the problems occur so that changes can be made to the road network.

Stress Testing

Stress Testing explores the issues of what happens to the system as it is put under valid load in excess of it design parameters. It is sometimes defined as to break the system but this is not the point of the test, though it may be the result. The purpose is to put the system under a number of very high load patterns and observe what happens to it. Some of the observations would be if the system just slows down, come to a stop, or causes serious damage.

For a road network this would be the equivalent of putting in a lot more vehicles. Perhaps the school is having an exchange visit, there is a conference in town, or the shops are having a special sale which attracts out of town visitors. Each or several of these would put a stress on the network.

Robustness Testing

Robustness Testing in contrast to Stress Testing deals with the response of the system under various loads when resources, such as a dependent system, hardware, or network, become unavailable. Again the purpose is not to break the system but study the system’s response to the various resource failures against several load patterns.

The equivalent for the road network would be having roads collapse, schools or works places shut down, or the shops deciding not the open. The purpose is to see if everybody could still get around town.

Endurance Testing

Endurance Testing checks how stable the system is over a period of time. It applies a high but normal load pattern of transactions to the system for a continuous period. Checks are made during the run to see if any subtle errors build up over a period of time which can affect the performance or integrity of the system.

In road network terms this would be checking usage over several consecutive days to see what happens. It may be that vehicles break down and stop people getting around, or because of extended jams schools and shops do not open on time. These errors build up over a period of time.

Load Testing Tools

All these tests use a load testing tool to assist in running of the tests. The Acceptance Criteria and tests have to be designed by the testers, and then implemented in the tool. The tool itself has three key functions:

1. Provide a simulated load to the system which can vary with time.
2. Record the results of running test loads.
3. Provide tools to display the results of tests.

In road terms they would be providing more vehicles to go on runs within the network, having watchers to count vehicles and record what is going on, and then writing a report on what is found.

When to Performance Test

• Types of Testing employed during a project including when performance testing takes place.
• User Requirements - three areas to include, such as non-functional requirements, and one to avoid when writing user requirements.

 

Part-2

Before starting the Performance testing, performance test engineer should collect non-functional requirement and Below is comprehensive list of performance Testing Requirements.

•  Total user base : How many users will have access to this application? 
• Customer will provide answer, how many users are having an access to the application.
• If the application is existing one, Customer will get the data from previous years to understand how many users are using the application. There are third party tools like Web trends to analyze the traffic of existing used application
• If the application is new and not yet in live, Business analyst and Application owners will be able to provide total user base.
• Concurrent User Load :What is the number of total users who will access the application together at any given point in time?
• According to industry standard, it is 10% of total users will be the concurrent load but this formula will not applicable for all the domains.
• Third party tools like web trends will be useful
• Business Processes ( Performance Scenarios):  
• What are the business processes (Workflows) to be performance tested with concurrent users? Objective here is to find few business processes which can cover up to 80% of the application code"
• 
  
• Load Distribution :What is the user/Load distribution on each business processes  listed 
• What is the time duration taken for each business process to complete end to end(Workflow)? 
• How often are the users doing each business process (Workflow) in an hour, a day, etc? to get this data for existing application
• Service Level Agreement (Performance - Service Level Agreement):
• What are the acceptable response times for each transaction (page) in all the business flows that needs to be performance tested? Mention SLA for against each of the transaction (page) identified
• 1. All pages within the portal shall load in 3 seconds or less.
• 2. Report should load in 5 sec
• 3. Yearly Report should load in 20 
• 4. SecgetAccountHistory API’s roundtrip response time should be < 250 msec

Performance Test Plan

Performance test plan are backbone of performance testing and they set the path to drive through whole testing process, It should clear all doubt roaring our mind about performance test like what to test, how to test, what is the criteria. A good test plan drives the project to achieve it's goals without any major difficulties.
While working in IT industry and for different vendors you may see changes in the test plan and every company maintain their our Test plan.
As mentioned test plans changes with the companies but their structure and basic need and fields remains same everywhere,It will portray answer for each question for our testing purpose.


A test plan should posses:


1. Document Information/Objective of the test


This section provided the high level summary of the need for this performance testing, It provided summary about test plan as well what all sections will be covered within the test plan
There are two basic parts for this section

• About the document
• Document Approval/Sign off

 2. Application Information


This section contains following below information

• Application Name
• Application Description
• Application Platform/Run-Time environment
• Application flow
• Application/System technology employed

 Diagramatic representation will give better picture from end user's oerspective.

3.  Key Business function definition

This section will provide information about what all business functions are in scope and need to be tested.

4. Performance test workload model

User / Business Transaction based workload will be given under this section

5. Test Scenario

Contains what type of test scenario will be performed as part of this testing effort. e.g. Average, Peak Hour, Soak etc.

This section also contains the information about ramp-up, ramp-down, think time as well.


6. Test User Profiles and IDs


This section the various types of user profiles that will be needed to test the application are defined, Laso the roles and entitlements of the test IDs will be described


7. Test Data


Test data id the base for any performance test, It could be nay form like URL, User IDs/Password and application specific data, this section described all types of data going to be used and where we can get the data from, Data is also provided by the data team and it should be mentioned how to take data ot the methods to process the data.


8. Test tools


Tools which will be used in testing will be descibed here with their versions


9. Test Schedule


Time line given to complete the testing will be described in this section, though its hard to calculate time lines but tentative periods should be defined in this section.


10. Issue tracking process


How the issues and bottlenecks found in the testing will be tracked and what tool or methodology will be used to track them.