Guidelines: Important Decisions in Test

Decide which artifacts to use and how to use each of them. The following table describes which artifacts are mandatory, and which can be used in some cases. The 'tailoring comments' give some guidelines to how you can tailor each artifact.

For a complete description of each artifact, see Test Artifacts. For each artifact read the section "Tailoring" where the advantages and disadvantages of that specific artifact are discussed.

Artifact	Tailoring Comments
Test Case	Mandatory. The question is to what extent are you going to create them.
Test Procedure	Mandatory. This artifact is the actual test - the question here is not if, but how and what kind of structure (format and detail) you are going to use.
Test Script	Optional (Mandatory for automated testing). Based upon your structuring of the test procedures, determine if the test scripts will implement and execute entire test case(s) (or use case) or will need to be combined with others to completely implement and execute a test case(s) (or use case).
Test Classes in the Design Model	Use this, if necessary, when you develop the test components.
Test Packages in the Design Model	Use this if you need to organize the test classes.
Test Components in the Implementation Model	Mandatory. You will always develop components for testing.
Test Subsystems in the Implementation Model	Use this if you need to organize the components.
Defects	Mandatory. You will find defects. The issue is to decide how the defects should be handled.
Test Plans	You need a test plan, unless the testing is trivial.
Workload Model	Optional (Mandatory for Performance Tests). The structure (format and content) of the document is what one should consider as being configurable.

The next sections give some guidelines to help you decide how you should use the test artifacts.

Test case

Test cases are created by the test team and usually treated as "Informal," that is approved by someone within the test team. This is a natural decision, especially when members of the test team participated in the requirements workflow.

If the test cases are developed by testers that have not been part of the requirements workflow, then the test cases are usually informally approved by project team members who participated in the requirements workflow.

If a customer wants to validate a product before it is released, some test cases could be selected as the basis for that validation. Treat these test cases as "Formal-External."

Test procedure

Test procedures should be treated as "Informal" artifacts - approved by someone within the test team and under configuration management control.

When test automation or external review is required, they should be treated as "Formal- Internal" or "Formal-External."

Test Scripts

Test scripts are usually treated as "Informal," that is approved by someone within the test team.

If the test scripts are to be used by many testers or shared (reused) for many different tests, they should be treated as "Formal-Internal."

Test artifacts in design and implementation

There are test artifacts in the design model; test class and test package, and the test artifacts in implementation model: implementation subsystems, and components.

These artifacts are like design and implementation artifacts - but created for the purpose of testing. The natural place to keep them is together with the design and implementation artifacts.

Remember to label them in such a way that they are clearly separated from the design and implementation of the system.

Defects

Defects are usually treated as "Informal" and usually handled in a defect-tracking system. In a small project you can manage the defects as a simple list, for example using your favorite e-mail system. But, this is only manageable for small systems, when the amount of defects grows, you need to start using a defect-tracking system.

Another decision to make is whether you need to separate the handling of defects (symptoms) from faults (the actual errors). In small systems you may manage to keep track of the defects only, and implicitly handle the faults. However, as the system grows, you must separate the management of defects from faults. For example, several defects may be caused by the same fault. So, if a fault is fixed, it is necessary to find the reported defects, and inform the users who submitted the defects. This is only possible if defects and faults are managed separately.

Test Plan

In any project where the testing is non-trivial, you need a test plan. In many cases it is treated as "Casual," i.e. not reviewed and approved. If testing is important, it could be treated as "Formal-Internal" or even "Formal-External."

Identify Configuration Items

Read Test Artifacts and decide which of the artifacts should be handled as separate configuration items.

It is up to each project to decide what needs to be handled as separate configuration items. It is basically a matter of how you want to control versions of artifacts separately. Therefore consider how you will work with the test artifacts.

The following sections give guidelines on configuration management of some test artifacts.

Test Cases

Handle test cases in your CM strategy the same way you would any other artifact. But do not handle the test cases together with the corresponding use cases as single configuration items, because they have their own life-cycles.

Test Procedures

Handle test procedures in your CM strategy the same way you would any other artifact. But do not handle the test procedures together with the corresponding test case(s) (or use cases) as single configuration items, because they have their own life-cycles.

Test Scripts

Most test tools treat test scripts as regular files, so each test scripts can be a configuration item. It is best to keep those test scripts that together make up the regression test for the system, in one configuration for each iteration. Use this configuration to hold all the different test scripts (configuration items) together to perform regression tests in the next iteration.

Decide on Test Approval Criteria

You must decide who will be responsible for determining if an iteration has met its test criteria. This individual or group will then decide if the iteration has met its test criteria, if all tests were completed successfully, and if the system fulfills the stipulated criteria.

This is a important decision, you cannot reach a goal if you do not know where it is. Note that for early iterations it may not be necessary that the tests pass, only that they have been identified, developed and run.