Tool Mentor: Using TestManager^TM to Review and Evaluate Requirements-based Test Coverage

Purpose

This tool mentor describes how to use Rational TestManager^TM to review and evaluate test coverage based on use cases / requirements.

Related Rational Unified Process activity:

• Section "Review and Assess Test Coverage" in Activity: Design Test.
• Section "Create or Acquire Test Scripts" in Activity: Implement Test.
• Section "Evaluate Test-Case Coverage" in Activity: Evaluate Test.

Overview

This Tool Mentor is applicable when running Microsoft Windows 95/98/NT 4.0. 

TestManager lets you create and run three requirements-based test coverage reports, each with a similar format, for:

• Test Planning Coverage - displays the number and percentage of test requirements for which test scripts have been assigned (planned)
• Test Development (implementation) Coverage - displays the number and percentage of test requirements for which test scripts have been created (programmed or recorded)
• Test Execution Coverage - displays the number and percentage of test requirements for which test scripts have been assigned (planned); and the number and percentage of test requirements for which test scripts have been played back.

Creating and running these reports includes:

1. Create report(s).
2. Run report(s).

1.   Create report(s)

To create any of the requirement-based test coverage reports:

1. Start Rational TestManager.
2. Click Reports > New > [report type] Coverage (where report type is one of the three coverage reports).  Alternately right-click the [report type] Coverage folder in the Asset Browser and click New Report. This displays the Coverage Report window.
3. Type a description for the report.
4. Set the Display options:

Indentation scale – Lets you specify the number of spaces to indent each sub-level in the report. Specify a number from 0 to 10.

Initial number of levels – Lets you specify the initial number of levels in the Requirements Hierarchy to be displayed in the report. Later, when you run the report, you can collapse or expand the hierarchy .

5. Set the following Data options:

To include the following data in the report, check the appropriate box.

NOTE:  The options listed below vary depending on the type of report being created.

Planning Coverage Report

Script names – The report includes a column showing the names of the scripts planned for the items in the Requirements Hierarchy.

# Scripts planned – The report includes a column showing the number of scripts planned for the items in the Requirements Hierarchy.

% Requirements covered – The report includes a column showing the percentage of planned scripts for hierarchy items that have been recorded using Robot.

Coverage goal – The report includes a column showing if the number of planned test scripts meets the specified coverage goal. You can set the goal by typing a number or by using the spin buttons.

Development Coverage Report

Script names – The report includes a column showing the names of the scripts planned for the items in the Requirements Hierarchy.

# Scripts planned – The report includes a column showing the number of scripts planned for the items in Requirements Hierarchy.

# Scripts developed – The report includes a column showing the number of scripts recorded with Rational Robot.

% Scripts developed – The report includes a column showing the percentage of planned scripts that have been recorded with Robot.

Developed goal – The report includes a column showing if the number of recorded scripts meets the specified coverage goal. You can set the goal by typing a number or by using the spin buttons.

Execution Coverage Report

# Scripts planned – The report includes a column showing the number of scripts planned for the items in the Requirements Hierarchy.

# Scripts executed – The report includes a column showing if the scripts planned for each requirement have been played back with Robot.

% Scripts executed – The report includes a column showing the percentage of planned scripts for each requirement that have been played back with Robot.

Execution goal – The report includes a column showing if the number of executed scripts meets the specified execution coverage goal. You can set the goal by typing a number or by using the spin buttons.

Script names – The report includes a column showing the names of the scripts planned for the items in the Requirements Hierarchy.

# Script runs – The report includes a column showing the number of times scripts have been played back for the items in the Requirements Hierarchy.

Script results – The report includes additional columns showing the number of scripts that have passed and failed for the items in the Requirements Hierarchy.

% Script runs passed – The report includes a column showing the percentage of scripts that have passed for the items in the Requirements Hierarchy.

Pass goal – The report includes a column showing if the number of passed scripts meets the specified coverage goal.

6. Select a script query from the list. Use the query to narrow down the number of items to be displayed in the report. Optionally, click Edit to modify the selected query, or click New to create a new query.
7. Click Save, type a name for the report, and click OK.

The report just created may now be run.  See Run the report(s) below for details

2.   Run report(s)

Once you create the reports, you can run them.  To run any of the three reports:

1. If the dialog for the desired coverage report is open, click Run.

If the dialog box is not open, click Reports -> Run to open the report list,  select the desired coverage report and click OK.

2. Select the requirements you want to include in the report. Your choices are as follows:

All requirements – Includes all requirements, regardless of type, in the Requirements Hierarchy. With the full version of RequisitePro, you can define multiple requirement types. With the baseline version of RequisitePro that is included with TestManager, there is only one requirement type.

All requirements of type – Includes all requirements of the selected requirement type.

Specific requirement of type – Includes a single branch of the Requirements Hierarchy for a specific requirement type. In this case, a branch is a single parent requirement and its children.

3. Click OK

NOTE: If running the Execution Coverage Report, an additional dialog box will appear.   Select the test results log(s) which you wish to include in the report and click > to add them to the Selected Log list.   Click OK to run the report.