REVIEW: Load and Performance Testing using WAPT

(Cross Posted from Learn QTP Blog.)

This is a guest post by Ivan Lisitsyn. He is an experienced load testing specialist. He works in SoftLogicacompany for the last 10 years and is its own Steve Jobs for WAPT.

Please let me introduce WAPT, a load testing tool for web sites and any applications that have web interface. In this article we will review the product features and will try to understand what makes WAPT special. But let’s start with a more general question.

When you need a load testing tool?

We all know that any software product requires a careful testing before it is ready for the delivery to the end users. If the product is a complex system consisting of several components, you may need to have different tests for each of them. In addition you may need to design tests to check the integrity of the system. In all such test you check the functionality. You provide some input data for your application and check that the output is correct. This is called Functional Testing. However if the tested product can be accessed by multiple users simultaneously, which is true for any web application, functional testing is not sufficient. Moreover, your web application may have successfully passed all functional tests, which means that it works perfectly for one user. Practice shows that in such case there is only a 50% chance that it will continue working correctly when accessed by multiple users. The more users you add, the greater probability that it will fail. That is why Load Testing is so important. Here are three important facts about it.

• Load testing is always performed as the last testing step, which means that the application should pass all functional tests before that stage.
• Load testing deals with quality requirements, which should be defined by your business objectives. After the testing you can compare actual performance parameters with the desired ones. If you do not have the exact requirements, you cannot tell if the application passed the test or not.
• Load testing cannot be performed manually without testing tools that emulate multiple simultaneous users accessing the target application. That is why the rest of the article makes sense.

Efficiency approach

A good functional testing tool should be able to create a very accurate emulation of real user behavior. It should also have a set of features to check that the tested application works correctly. In case of a load testing tool you have to require something more on top of that: efficiency. When you need to create thousands virtual users to test your web site, it will be a bad surprise for you if your testing solution can create only a hundred users with help of one server. This will increase your hardware expenses dramatically. Fortunately, there is another difference which makes the problem a bit easier: load testing deals only with the server part of the tested application. It does not check that the client part works correctly, so it does not need to emulate a real user working with the web interface. It can emulate the work of that web interface and communicate directly with the server. In any case the tests are initially created by recording user actions. However functional tests consist of such actions like mouse moves, keyboard strokes, clicking buttons on specific windows, etc. Web load testing tools usually record tests in form of sequences of HTTP requests sent to the server as a result of user activity in a browser. These requests contain all the session-specific data that client sends to the server. The responses carry data in the opposite direction. This approach is much more efficient, because it does not require launching a copy of the client application to emulate each user. As a result, a wisely designed load testing tool, such as WAPT, can create thousands virtual users with help of just one computer. This makes it different from functional testing products extended with additional load testing modules. This is the reason why it is normal to have a separate product for load testing even if your current testing solution claims to include that functionality.

Do you have to be a programmer to create load tests?

In fact, the above mentioned approach is not completely new. Efficient load testing tools appeared many years ago, soon after first web sites. The general problem with those tools was that to create a test you had to write a program in some scripting language. The execution of that program by the testing tool actually produced the desired load on the target web site. If you wanted to change any test parameter, such as the number of virtual users, you had to find the corresponding value in the script and modify it. For some reason this approach in load testing appeared to be very persistent. Even in our days most test engineers refer to “test scripts”, if they want to name something containing test specification. Moreover, even though most load testing tools can record tests with just few clicks and change test parameters through a convenient GUI, they still keep the real test specification in a script. WAPT is completely different in this respect. User behavior on the tested web site is specified in a so-called virtual user profile. In most cases this is just a sequence of HTTP requests which will be sent one by one to create a copy of the initially recorded user session. If you need to make different users do different things, you can create several profiles, or you can add operators that will switch the execution depending on session-specific conditions. In any case the profile will be represented by a tree-like structure where you can edit the properties of each node. This is a bit more complex than a sequence of requests, but still much easier to work with than a script.

Accurate real user emulation

How is it possible to emulate the behavior of real users without running the client application? Obviously each user session can include some specific data, which cannot be the same in other sessions. This is not just a session ID, which is usually handled by cookies. You may need to make different virtual users use different names and passwords. Moreover, during the user session the client application may receive some data from the server and use that data in subsequent actions. All such problems can be resolved by proper parameterization of HTTP requests. In WAPT you can see and edit the list of parameters for each request when you observe its properties. If you need to make some parameters changing for different sessions, you can easily do this. There is a wide set of functions for this purpose. For example, you can read user names and passwords from a file one by one for each new session. If you need to extract a value from server response, you can define a variable and assign the extracted value to it. In subsequent requests you can use that variable to specify the value of a parameter. What is even better, WAPT automates that process for the most common cases. For example, if it finds a hidden field inside an HTML form, it automatically creates a variable for it. The above approach is good to handle about 95% of all tests. However what if you really need to implement a complex calculation, which is normally performed by the client part of your web application? You can do this with help of JavaScript code that you can insert between any two requests. So, WAPT actually has all the same capabilities as the script-based tools. However it is much easier to use in less complex cases. In addition to the accurate emulation of user session, WAPT can also easily check if the server response is correct or not. Usually this can be done by searching a certain keyword inside the server response. The word “error” is a popular one for this. However, if you want a more complex processing, you are welcome to use JavaScript.

Let’s specify the load for the web site

Once you created profiles for all types of virtual users that you want to add to the test, you need to specify the load parameters. WAPT provide a good flexibility in this respect. First of all, you can choose one of three types of load for each profile: constant, ramp-up or periodic. In each case you can adjust corresponding load parameters. For example, for growing load you can specify how many virtual users should be added at each step and how long that step will take. If you use several systems to generate the load, you can assign certain profiles to specific load agents. For example, if you run a distributed test and one of your load agents is located in Australia, you can create a special profile for Australian users and assign it to the load agent located there.

Test results analysis

As soon as all load parameters are specified, you can click the “Run test” button and see what happens to your web site. WAPT starts delivering performance data almost immediately after you launch the test. You can see on charts how the error rate, response time and throughput parameters change as the load grows. In addition to charts you obtain a comprehensive report with about 20 different tables where you can see averaged data for each test phase. It is very detailed, so even if you need to measure response time for a particular request, you can find that information in the report. In some cases you may discover that your web site experiences problems only in certain specific sessions, or you may need to have the exact output to research the reason of produced errors. For this purpose you can enable logging and see all the requests and responses for each user session after the test completion. Since logs can occupy significant space, there are additional options that let you enable them only for the first users and limit the response length stored in logs. Finally, you can get data directly from one or several servers through SNMP interface for general system parameters, such as CPU usage. Database performance parameters can be obtained through an ODBC connection. A set of most useful counters for Oracle, MS SQL and MySQL is already included in the product, but you can also create your own ones. This data is included in the report along with the other performance information.

Is it applicable for any web site?

WAPT can be used to load test almost any web application. The only important note is that the application should be HTTP-based, which means that it should use HTTP protocol for client-server communication. Otherwise WAPT will not be able to record such communication and reproduce user sessions during the test. In fact, the above condition is not very limiting. Probably only about 1% of client-server applications, which also call themselves “web clients” do not meet that criterion. This is because “web” essentially means something accessible over HTTP. Only some sophisticated Java-based client applications which are not browser-based at all can use their own protocol instead of HTTP. Inside the HTTP field there are many specific things supported by WAPT: secured HTTPS connections, client certificates, proxy servers, a number of different authentication schemes, and much more. There is a bunch of small features which can become very important for a certain web site or a certain test case: browser cache emulation, streaming requests (continuing on the background while the virtual user executes the rest of the session), rendezvous point to synchronize the behavior of different users, IP spoofing to emulate users with different IPs, etc. In addition to the above the functionality of the product can be extended with several modules, each of which facilitates work with web sites utilizing a specific technology. For example, to facilitate the work with ASP.net sites you can install such a module and it will automate parameterization of view states. Currently the following modules are available: ASP.net, JSON, Flash/Flex, and Silverlight.

WAPT product line

WAPT is quite affordable in terms of prices. It costs less than most of its competitors, especially if you need a scalable solution for a high volume test. Unlike many other vendors, we do not charge for licenses depending on the number of virtual users that you need to create in your test. Instead of this we provide different versions and extensions that are required only in certain cases. The basic version of WAPT is not scalable and can use only one system both for the test design/management and for the virtual user emulation. The professional license allows you to install the workplace component (GUI) of the product on a single system and use up to 2 systems for load generation. If you want to extend this limit, you can purchase additional Load Agents. These agents are used to create greater test load for your web site. Each Load Agent can generate about 2000 virtual users in an average test. Some other useful features like direct server and database monitoring and JavaScript support are also available only in the Pro version. Finally, if you need to generate a very big test volume (over 10,000 virtual users), you can replace regular load agents with a powerful thing called x64 Load Engine.