Performance Testing – How Repeatable is your Test?
03 January 2022
It doesn’t matter if you are new to LoadRunner or Performance Center or if you are a seasoned user — understanding the sequence seed is important. What is the purpose behind randomizing values in a load test? What is the random sequence seed in LoadRunner / Performance Center? These are questions students often ask in LoadRunner and Performance Center training classes.
People are not robots
When designing a performance test, a big challenge is to have one single script that behaves like a group of real users. Real users are not robots. Real users are human — who move at different speeds and select different values. For example, I walked into a retail store the other day to purchase a new pair of tennis shoes for my son. Unfortunately, they didn’t have his size and color selection in stock. So, I decided to place an order online while in the store and have the shoes shipped directly to my house. Rather than having the salesman go through the painstaking effort of typing my contact information, I asked him if I could type the information and proceed online. Being a rather fast typist, (okay, I am a rather experienced online shopper as well) I completed entering all of the information and submitted the order in approximately 1.5 minutes. While some users may take 1.0 minute to complete this task, other users may take 3.0 minutes, yet others may take 4.5 minutes. Randomizing the think time values is an important step to getting that one single script to behave like real people instead of robots.
The importance of the sequence seed
When you complete the think time, pacing values and parameterized values that can be randomized; there is one additional setting to consider — the random sequence seed setting. By setting the sequence seed before running your test and using the same sequence seed value for a second test, you are using a mathematical technique of variance reduction. When you simulate the performance of two different systems, there are basically two sources of variation between them:
- Differences in the performance of the two systems
- Differences in the (randomly generated) workload
If you can force the simulation to use the same workload for both cases, you have eliminated one of those sources by testing the performance of the two systems against the same workload. This will increase the precision of the estimates that can be obtained for a given number of iterations.
After executing your baseline test, a piece of code is identified to be a performance issue. The normal development cycle occurs, and the modified code ends up in your environment, ready for another round of performance testing. Setting the sequence seed before executing the test again enables you to use the internal LoadRunner capabilities, allowing you to repeat random sequences of numbers.
Maximize the value
of your ALM
Govern quality and implement rigorous, auditable software lifecycle processes.
The secret of duplication
Beware that any modifications to the test, such as: modifying the values in the parameter file that are randomly selected, think time values, adding a new business process flow, etc. will defeat your attempt to duplicate the workload from the baseline test. In order for the variance reduction technique to be properly applied, the same sequence of events with the same randomization must be identical between the runs. If you choose not to apply a sequence seed to a performance test, a different pattern of random sequence seeds could occur — potentially resulting in a better (or worse) set of response times.
Also, when using the sequence seed and relying on its capabilities to repeat the same sequence of values, consider the potential in your simulation for system performance to influence the workload. You also need to consider the potential for system performance to influence the workload. You also need to consider the potential for system performance to influence the workload when using the sequence seed and relying on its capabilities to repeat the same sequence of values.
As a simple example, a higher performance system might handle a number of transactions in a fixed time test. If during the baseline test, you generate a total of 1,000 transactions, but during a subsequent test, after code enhancements, you generate a total of 2,000 transactions, only the first 50 percent of the values were reproduced during the second test. Although the same random number seed was used for both runs, the workloads the two systems were tested against were different. Depending on the random numbers generated in the second half of the test, you may over or underestimate the impact of the performance improvement over the baseline performance.
By using the power behind LoadRunner’s capabilities and setting or not setting the Scenario Run-Time Settings value of the random sequence seed as appropriate, you increase your ability to analyze and understand differences between the baseline and subsequent test results.