The problem: performance of large-scale distributed systems

The unprecedented growth of networking technologies created an extremely fertile ground for the new generation of application software systems - systems that offer their services to hundreds, thousands, and even millions of users.

Of course, performance became one of the most critical issues for the software systems of this kind, and developers required effective tools to solve performance-related problems of large-scale distributed software. The market responded to this demand with a number of automated performance testing and monitoring solutions. Many of these solutions have been available for years, yet they still do not address well the following key issues important for many categories of customers:

• Proprietary protocols. Most performance testing tools cannot be adapted to support an arbitrary protocol at all. Those that can be adapted usually impose too many constraints on how adaptation can be done: dictate usage of a certain programming language, support only one operating system, etc.

• Profiling under load. Many performance testing tools do not provide any ways to profile the tested system and reconcile the profiling results with "external" performance measurements and other relevant data. Other tools allow to do some profiling only using OS-specific services, such as performance counters in Microsoft Windows. However, designed rather for performance monitoring, these services are very inconvenient and difficult to use for really comprehensive profiling.

• Scalability. Although some of the performance testing tools scale quite well when dealing with specific protocols, scalability of customized solutions is hampered by certain limitations of extensibility mechanisms (for example, by creation of new threads or processes for each simulated user or client).

• Peer-to-peer communications. Most performance testing tools are well suited only for client/server communication model: they generate load by simulating activities of independent clients. It is either impossible or extremely difficult to apply such tools to peer-to-peer or other communication models.

These and other issues often force developers to create their own, proprietary solutions for performance testing and profiling. Such home-grown solutions have a serious drawback: as they grow, substantial part of the development and maintenance costs inevitably becomes associated with general testing infrastructure, and not with project-specific test logic. Many owners of such home-grown solutions would be happy to migrate their existing testing code to a third-party test automation platform, but this is not easy to do even with the most advanced of existing incumbent load testing products that were not designed for this kind of customer.

The solution: distributed testing automation framework

We tried to address the issues mentioned above and developed a product that provides an open framework for creating customized solutions for automated load testing.

XtremeLoad can be used to distribute almost any kind of impact simulation, performance measurement, performance monitoring, and performance profiling software to any kind of computer or intelligent device, as long as the appropriate custom test tool has been targeted to that computer or device. This approach enables truly comprehensive performance testing of complex distributed software/hardware systems - from end-to-end testing of user's experience to testing of individual components on implementation and integration phases of software development.

XtremeLoad features:

• Rich set of common test automation services. The framework provides all the key services necessary for any kind of distributed load test automation system: single-point control of all computers participating in tests; advanced scheduling of tests on shared pools of computers; automatic deployment (and, when necessary, updating) of test logic (test cases) across test lab; automatic consolidation of results gathered on multiple computers in a central repository (SQL database); powerful tools for graphical analysis of test results and reporting.

• Out-of-the-box support of performance profiling. If you are developing a complex multi-tier system and want to profile it under load, all you have to do is instrument your code using simple XtremeLoad profiling API. The framework will do the rest: it will deploy a special test tool to the computers hosting your system, it will collect the profiling data recorded by your system via profiling API, and finally it will let you reconcile those profiling data with all other results, such as external performance measurements, obtained during test execution.

• Layered extensibility APIs for Java and Win32. XtremeLoad's extensibility interfaces - APIs that can be used to create custom load testing solutions - are offered for both Java and Win32. That provides a customer with practically unlimited choice of technologies that can be used to implement a custom solution. Java APIs also provide very wide portability of the custom test tools across operating systems and hardware. Finally, different layers of extensibility APIs give a very important freedom to a developer of custom solution - a freedom to make her own, most suitable for a given project, tradeoff between implementation simplicity and functional/performance capabilities of the custom test tool.

• Out-of-the-box support of general-purpose performance tracking. XtremeLoad provides a specialized test tool that can be used to record any performance counters - system-level or application-specific ones - available on any computer running Windows operating system.

In order to demonstrate a turnkey solution that can be built on XtremeLoad platform, we developed an advanced Web application test tool. Combined with other tools available out-of-the-box (see above), this test tool provides a truly comprehensive load testing solution for developers of complex multi-tier Web applications. It not only allows to simulate activities of Web users and measure their perception of application performance, but also enables efficient functional testing under load and profiling of the server-side components of a Web application.

The benefits: what XtremeLoad can give you

XtremeLoad provides substantial benefits - savings of development cost and time, as well as improvement of product quality - for several classes of customers:

• Corporate customers get an effective instrument to solve performance problems not just find them.

  Almost any load testing tool can do an excellent job of locating where a complex web application performs poorly, or pinpointing where functionality in that application fails under a heavy load. Unfortunately, that is the relatively trivial part of the problem. The other more difficult part is for the application's developers to profile and debug the internal application logic, to pinpoint exactly why the application fails to perform well or behave correctly.

  XtremeLoad offers simple-to-use APIs that allow to gather any relevant performance data from inside the application and reconcile those data with all other results obtained during the load test.

• Customers with proprietary protocols get a more flexible and more scalable way to automate performance and functional testing.

  A proprietary protocol may be much more complex than a stateless sequence of request/response interactions. A proprietary protocol may use several other protocols at its transport level, such as when a wireless messaging device needs to interface to proprietary protocols within the telecommunication provider's network. A proprietary protocol may also be hard to model as a simple client/server interaction, such as with many instant messaging protocols. Finally, a proprietary protocol may be deployed so widely that load testing of the system would require simulation of thousands or millions of concurrent users. Issues like these make it very difficult, if at all possible, to load-test proprietary protocols using existing tools focused primarily on the standard, mainstream protocols and APIs.

  Making it easier to create custom test automation solutions that scale was one of the key initial goals of our XtremeLoad project. XtremeLoad's extensibility APIs were designed to strike a best possible balance between easiness of building custom test tools and a technological freedom of how such tools can be implemented. While implementation of general-purpose services and infrastructure plumbing are offloaded to the framework, a custom test tool developer can use nearly any programming language, architecture, and technology to better address the issues specific for a given protocol or project.

• Customers with home-grown testing solutions can re-use their investment in existing code.

  The vast majority of the home-grown solutions are implemented using the owner's preferred programming language. This very natural situation can quickly become a serious impediment for migration to a third-party test automation tool or framework that dictates other, sometimes even proprietary, languages to be used for test logic implementation. Even when programming language is not an issue, migration may be difficult and expensive because of more subtle architectural or technological problems (e.g. problems related to multi-threading, asynchronous I/O, etc.). All the gains from migration to a new tool or framework may be destroyed by the costs of retraining available testers or hiring new ones.

  XtremeLoad was designed with this problem in sight. Its extensibility APIs impose only minimum technological constraints on implementation of custom test tools. As a result, XtremeLoad lets the customers reuse maximum of the past investments into performance test automation - especially investments into custom, project-specific or product-specific testing logic.

Interested?

Currently XtremeLoad is under development, nearing its first public beta release.

For additional technical or commercial information about this product, please send email to support@uscsw.com or sales@uscsw.com.

Products|Services|Company|Contacts