Sören Janstål

Data Research DPU ab Data Research DPU for Evaluation of Information Technology

Sören Janstål

Ovum Evaluates: Software Testing Tools

editor Graham Titterington, manager Eric Woods

Svenska

Se order form for price!

Software failure is not just expensive. It can spell disaster for your business.

Guarantee your performance with the definitive guide to software testing tools from Ovum.

• Continuously updated advice on tools, vendors and market developments
• Expert evaluations of all the leading testing tools from your perspective
• Advice on the latest developments in Web application testing.

How much will software failure cost your business?

No business can afford to lose hours of trading while its server is brought back on line, and no organisation can afford to lose its phone system for hours at a time. But without adequate testing of your software, the risk of failure is very real. Manual testing is not enough; to be sufficiently reliable, software testing needs to be automated.

Advances in e-commerce, and intense competition in the Web marketplace, are driving a new demand for automated testing of Web-based applications. The lack of human "sanity checks" to monitor business transactions over the Web, coupled with the speed and volume of Web business, means that any error can cause financial loss that is both immediate and crippling.

Ensuring that your business doesn’t suffer like this is more urgent than ever. But with tight deadlines, a vast array of testing tools to choose from, and a rapidly evolving product market, finding exactly the right product is not easy.

Ovum Evaluates: Software Testing Tools makes sure that you have all the information you need, when you need it. We enable you to evaluate the technology, the tools and the vendors and make the right choices for your organisation.

 Make informed decisions with Ovum's expert advice

Ovum Evaluates: Software Testing Tools compares the merits of the leading testing tools as soon as they are released or updated. We address the testing needs of all types of computer architectures, including mainframe, client-server, embedded, Web and GUI-based applications. Ovum will help you choose the tool which best serves your needs, and ensure that the tools you buy will not become shelfware.

Continuously updated, Ovum Evaluates: Software Testing Tools provides independent advice on the current testing tools market and expert forecasts of market development over the next 12 months.

What are the challenges of Web testing?

Web applications have brought a new dimension to software testing, and with it, new challenges:

• The constant evolution of the Web makes regular re-testing of applications a necessity.
• Web applications need to run successfully on many different browsers. So tests need to be repeated on each browser.
• Web applications can be updated quickly, without the need to distribute and install copies on every host. But software developed quickly is often bug-ridden unless thoroughly tested.
• Software performance is vital to retain your customers on the Web. But overloaded servers often refuse service. Can you be sure your software is handling these refusals correctly?
• The Web hosts an enormous range of technologies, including links, downloaded programs, and dynamic content, each with its own testing challenges. Many applications will use these kinds of content to deliver critical business services.

All these factors make automated testing a necessity for e-commerce, and pose challenges to the testing tool suppliers. We monitor how they are responding to these challenges and what this means for you, the user.

Ovum's reference model: a solid framework for in-depth evaluation

Ovum Evaluates: Software Testing Tools bases its evaluations on a robust reference model. We assess the capabilities of each tool or suite of tools against seven core dimensions. We use this model to classify and grade the tools, providing a straightforward, consistent basis for comparison:

1. Test planning and management – Activities such as creating test plans, defining, controlling and tracking test requirements, and managing test scripts must be well planned and managed to ensure that the testing process is fully effective, especially when the application under test is large and/or complex. In this dimension, we evaluate tool capabilities for test planning, test storage, management and control, requirements traceability, integration, error tracking & report generation.
2. Requirements analysis and test case generation – We assess the ability of the tool to find errors in requirements specifications and to generate test cases from those documents for later use in the testing cycle. We consider tool capabilities for document interpretation and analysis, requirements validation, test case generation & reporting of analysis results.
3. Inspections and reviews – Inspections are a cost-effective process for finding errors and improving the quality of all products in the software development process, and can be a very useful part of the validation process. We consider tool capabilities for process management and reporting, organising review meetings & issue tracking and resolution.
4. Code quality – We assess each tool’s ability to read and analyse source code for the purpose of generating metrics, detecting coding errors and violations of coding rules, and testing for compliance with defined coding standards. We consider code clarification, code checking, metrics determination, report generation & memory leak detection.
5. Functional testing – Functional testing of programs is often the first topic that comes to mind when the phrase "software testing" is mentioned. We cover all stages of this process, evaluating in detail the tool’s abilities to capture the behaviour of existing applications, develop and modify test scripts, execute tests, evaluate test results, and assess the effectiveness of a set of tests.
6. Load and performance testing – Load testing requires the definition of a scenario in which large numbers of simulated users carry out tasks, mirroring how the application will be used in the field. We evaluate tool capabilities for performance testing, architecture independent load testing, load testing for client-server applications, performance deficiency diagnosis and reporting of load test results.
7. Web application testing – We bring together all the concerns in testing functionality and performance of Web-based applications. We consider Web site integrity analysis, functional testing, load testing of Web-based applications and validating integrity under varying loads & security.

Who will benefit from the service?

Ovum Evaluates: Software Testing Tools is essential for anyone concerned with software testing and quality, including:

IT managers who need support for the process of tool selection and purchase;

Testing professionals who need to maintain and enhance their knowledge of current testing issues and technologies;

Project managers who need to understand how the tools can help them with their project-specific requirements;

Quality managers who are responsible for introducing effective tools and releasing software of known quality;

Software developers who are seeking to improve the quality of their development process;

Testing tool vendors who require market and competitor product information, and who need to encourage advances in the development of new testing techniques and technologies;

Computer systems vendors who need to know which are the best products for their hardware platforms.

How does our service support you?

Ovum Evaluates: Software Testing Tools examines testing tools from your perspective. Our product evaluations present the tools in a context that any software developer, tester or IT manager will recognise. We give a clear verdict on each product, along with detailed facts that make tool comparisons easy and unambiguous. We provide support at every stage of the testing process, whether your are defining your organisation’s product requirements, assessing the strengths and weaknesses of individual tools or monitoring the development of the tool market. Benefit from:

• Summary – Providing an executive level summary of product strengths and weaknesses, along with a description of each product, its major components and future enhancements.
• Commercial viewpoint – Offering a description of the vendor organisation, company background, services and support for you as the customer, pricing structure and contact details.
• In-depth evaluation – Our continuously updated evaluations provide a core technical assessment of each product.
• Reference – Providing you with the facts you need to generate a "long list" of products to meet your testing needs, we describe the applications that each tool can be used to test and the environment in which the product operates.

Two options to suit your needs

For a quick decision...

 To make your selection now, you need the Decision Maker package:

Is your organisation embarking on the selection process for new testing software? Are you involved in making the decision? If you need to:

• Understand the technology and its benefits for your organisation;
• Draw up a shortlist without wasting months on "beauty contests";
• Make and justify the right product/supplier choice within strict deadlines;

then Ovum’s Decision Maker package is for you. You’ll receive:

All current product evaluations – we are continuously updating our evaluations of the leading products in the market. When you buy, you’ll receive a complete set of up-to-date evaluations, helping you to make a decision that is right for your current and future needs.

Guide to the evaluations – to help you get the most from our evaluations, we provide a detailed description of our methodology and what to look for in the products.

Everything you need to back up your decision – an overview of software testing: what is it, how do you do it and why is it critically important to your organisation’s future health?

To be always up-to-date on the latest testing software developments...
For on-going, up-to-the minute information from the heart of the testing tools market, you need Decision Maker PLUS+

If you need regularly updated product information and market news in order to:

• Understand which supplier/product best matches your requirements on a continuous basis;
• Assess how your company will benefit from testing software now and in the future;
• Identify the strengths and weaknesses of your own products and strategies against those of your competitors if you are a supplier;
• Assess the market opportunities in software testing;

then a subscription to Decision Maker PLUS+ is for you. In addition to all the elements of the Decision Maker service, you’ll receive:

Product evaluations – continuously updated for a whole year. As new releases become available, we’ll update our product evaluations to keep them current. We will also evaluate significant new products as they are launched. Subscribers to Decision Maker PLUS+ will receive additional evaluations as soon as they become available.

Annual market forecast – Ovum’s expert analysts forecast market development, predicting how sections of the market will grow or decline, and identifying the key factors behind this movement. Ovum’s Annual Forecast provides you with the insight you need to plan your business strategy.

Regular updates – Ovum Evaluates: Software Testing Tools will bring you regular updates on products and market developments to ensure that you stay informed about this fast-moving technology area.

Telephone enquiry service – speak directly to the Ovum Evaluates: Software Testing Tools experts for additional information and advice when you need it.

Current evaluations include tools from the following vendors:

Aonix	Compuware	Cyrano	IBM
McCabe & Associates	Mercury Interactive	Rational Software	Reliable Software Technologies
RSW. Segue Software	Sun Microsystems	Technology Builders

About the authors

• Bloor Research Group: CAST Tools: An Evaluation and Comparison
• Bloor Research Group: Testing Tools