(Cross Posted from Learn QTP Blog.)
This is a guest post by Ivan Lisitsyn. He is an experienced load testing specialist. He works in SoftLogicacompany for the last 10 years and is its own Steve Jobs for WAPT.
Please let me introduce WAPT, a load testing tool for web sites and any applications that have web interface. In this article we will review the product features and will try to understand what makes WAPT special. But let’s start with a more general question.
When you need a load testing tool?
We all know that any software product requires a careful testing before it is ready for the delivery to the end users. If the product is a complex system consisting of several components, you may need to have different tests for each of them. In addition you may need to design tests to check the integrity of the system. In all such test you check the functionality. You provide some input data for your application and check that the output is correct. This is called Functional Testing. However if the tested product can be accessed by multiple users simultaneously, which is true for any web application, functional testing is not sufficient. Moreover, your web application may have successfully passed all functional tests, which means that it works perfectly for one user. Practice shows that in such case there is only a 50% chance that it will continue working correctly when accessed by multiple users. The more users you add, the greater probability that it will fail. That is why Load Testing is so important. Here are three important facts about it.
- Load testing is always performed as the last testing step, which means that the application should pass all functional tests before that stage.
- Load testing deals with quality requirements, which should be defined by your business objectives. After the testing you can compare actual performance parameters with the desired ones. If you do not have the exact requirements, you cannot tell if the application passed the test or not.
- Load testing cannot be performed manually without testing tools that emulate multiple simultaneous users accessing the target application. That is why the rest of the article makes sense.
A good functional testing tool should be able to create a very accurate emulation of real user behavior. It should also have a set of features to check that the tested application works correctly. In case of a load testing tool you have to require something more on top of that: efficiency. When you need to create thousands virtual users to test your web site, it will be a bad surprise for you if your testing solution can create only a hundred users with help of one server. This will increase your hardware expenses dramatically. Fortunately, there is another difference which makes the problem a bit easier: load testing deals only with the server part of the tested application. It does not check that the client part works correctly, so it does not need to emulate a real user working with the web interface. It can emulate the work of that web interface and communicate directly with the server. In any case the tests are initially created by recording user actions. However functional tests consist of such actions like mouse moves, keyboard strokes, clicking buttons on specific windows, etc. Web load testing tools usually record tests in form of sequences of HTTP requests sent to the server as a result of user activity in a browser. These requests contain all the session-specific data that client sends to the server. The responses carry data in the opposite direction. This approach is much more efficient, because it does not require launching a copy of the client application to emulate each user. As a result, a wisely designed load testing tool, such as WAPT, can create thousands virtual users with help of just one computer. This makes it different from functional testing products extended with additional load testing modules. This is the reason why it is normal to have a separate product for load testing even if your current testing solution claims to include that functionality.
Do you have to be a programmer to create load tests?
In fact, the above mentioned approach is not completely new. Efficient load testing tools appeared many years ago, soon after first web sites. The general problem with those tools was that to create a test you had to write a program in some scripting language. The execution of that program by the testing tool actually produced the desired load on the target web site. If you wanted to change any test parameter, such as the number of virtual users, you had to find the corresponding value in the script and modify it. For some reason this approach in load testing appeared to be very persistent. Even in our days most test engineers refer to “test scripts”, if they want to name something containing test specification. Moreover, even though most load testing tools can record tests with just few clicks and change test parameters through a convenient GUI, they still keep the real test specification in a script. WAPT is completely different in this respect. User behavior on the tested web site is specified in a so-called virtual user profile. In most cases this is just a sequence of HTTP requests which will be sent one by one to create a copy of the initially recorded user session. If you need to make different users do different things, you can create several profiles, or you can add operators that will switch the execution depending on session-specific conditions. In any case the profile will be represented by a tree-like structure where you can edit the properties of each node. This is a bit more complex than a sequence of requests, but still much easier to work with than a script.
Accurate real user emulation
Let’s specify the load for the web site
Once you created profiles for all types of virtual users that you want to add to the test, you need to specify the load parameters. WAPT provide a good flexibility in this respect. First of all, you can choose one of three types of load for each profile: constant, ramp-up or periodic. In each case you can adjust corresponding load parameters. For example, for growing load you can specify how many virtual users should be added at each step and how long that step will take. If you use several systems to generate the load, you can assign certain profiles to specific load agents. For example, if you run a distributed test and one of your load agents is located in Australia, you can create a special profile for Australian users and assign it to the load agent located there.
Test results analysis
As soon as all load parameters are specified, you can click the “Run test” button and see what happens to your web site. WAPT starts delivering performance data almost immediately after you launch the test. You can see on charts how the error rate, response time and throughput parameters change as the load grows. In addition to charts you obtain a comprehensive report with about 20 different tables where you can see averaged data for each test phase. It is very detailed, so even if you need to measure response time for a particular request, you can find that information in the report. In some cases you may discover that your web site experiences problems only in certain specific sessions, or you may need to have the exact output to research the reason of produced errors. For this purpose you can enable logging and see all the requests and responses for each user session after the test completion. Since logs can occupy significant space, there are additional options that let you enable them only for the first users and limit the response length stored in logs. Finally, you can get data directly from one or several servers through SNMP interface for general system parameters, such as CPU usage. Database performance parameters can be obtained through an ODBC connection. A set of most useful counters for Oracle, MS SQL and MySQL is already included in the product, but you can also create your own ones. This data is included in the report along with the other performance information.
Is it applicable for any web site?
WAPT can be used to load test almost any web application. The only important note is that the application should be HTTP-based, which means that it should use HTTP protocol for client-server communication. Otherwise WAPT will not be able to record such communication and reproduce user sessions during the test. In fact, the above condition is not very limiting. Probably only about 1% of client-server applications, which also call themselves “web clients” do not meet that criterion. This is because “web” essentially means something accessible over HTTP. Only some sophisticated Java-based client applications which are not browser-based at all can use their own protocol instead of HTTP. Inside the HTTP field there are many specific things supported by WAPT: secured HTTPS connections, client certificates, proxy servers, a number of different authentication schemes, and much more. There is a bunch of small features which can become very important for a certain web site or a certain test case: browser cache emulation, streaming requests (continuing on the background while the virtual user executes the rest of the session), rendezvous point to synchronize the behavior of different users, IP spoofing to emulate users with different IPs, etc. In addition to the above the functionality of the product can be extended with several modules, each of which facilitates work with web sites utilizing a specific technology. For example, to facilitate the work with ASP.net sites you can install such a module and it will automate parameterization of view states. Currently the following modules are available: ASP.net, JSON, Flash/Flex, and Silverlight.
WAPT product line