Activity: Evaluate Test

Purpose To deliver quantifiable measures of testing progress. To generate a test evaluation report.
Steps Evaluate Test-Case Coverage Evaluate Code Coverage Analyze Defects Determine if Test Completion and Success Criteria have been achieved
Input Artifacts: Test Plan Test Cases Defects	Resulting Artifacts: Test Evaluation Report, completed
Worker: Test Designer

Evaluate Test-Case Coverage

Purpose

To determine if the required (or appropriate) requirements-based test coverage has been achieved

Tool Mentors:

Using Rational TestManager^{^TM} to Review and Evaluate Requirement-based Test Coverage

To evaluate test-case coverage, you need to determine:

The ratio between how many test cases have been performed in this iteration, and a total number of test cases for the target for test.
The ratio of successfully performed test cases.

The objective is to ensure that 100 % of the test cases targeted for this iteration have been executed successfully. If this is not possible or feasible, a different test coverage criteria should be identified, based upon:

Risk or priority
Acceptable coverage percent

See "Key Measures of Testing" in Introduction to Test.

Document the results in a Test Evaluation Report for this iteration.

Evaluate Code Coverage

Purpose

To determine if the required (or appropriate) code-based test coverage has been achieved.

To evaluate code coverage, you need to determine:

The ratio between how many lines of code that have been executed during test in this iteration and the total number of lines of code in the test target.

See "Key Measures of Testing" in Introduction to Test.

Document the results in a Test Evaluation Report for this iteration.

Analyze Defects

Purpose

To evaluate the defects and recommend the appropriate follow-on activity
To produce objective reports communicating the results of testing

Tool Mentors:

Using Rational ClearQuest^{^TM} to Analyze Defects

To analyze defects, you need to review and analyze the measures chosen as part of your defect analysis strategy. The most common defect measures used include three different measures (often displayed in the form of a graph):

Defect Density - the number of defects are shown as a function of one or two defect attributes (such as status or severity).
Defect Trend - the defect count is shown as a function over time.
Defect Aging - a special defect density report in which the defect counts are shown as a function of the length of time a defect remained in a given status (open, new, waiting-for-test, etc.)

Compare the measures from this iteration to the results from the analysis of previous iterations, to get a view of defect trends.

It is advisable to show the results in diagram form.

Determine if Test Completion and Success Criteria Have Been Achieved

Purpose

To determine if testing has been executed completely and acceptably
To identify the appropriate follow-on test activity
To produce objective reports communicating the results of testing

Review the test strategy defined in the test plan. It should present test criteria in terms of test coverage and, or defect evaluation. Examine the test results, defects, and the analysis of the defects, and determine if the criteria have been met.

If the criteria have not been met, there are several alternatives:

Gather additional information:

Produce different reports, such as different defect density reports
Investigate the process to determine if unexpected conditions contributed to the deviation from the stated test criteria and re-assess the criteria based upon this knew knowledge

Suggest scheduling of additional testing:

Implement new tests to add depth to the test cases
Implement new test to add greater test coverage

Modify the test criteria:

Review and assess the risk of changing the criteria POST testing
Identify a subset of the software that satisfies the test criteria and decide whether it can be deployed.