Concepts: Metrics
Topics
Why do we
measure? 
We measure primarily to gain control of a project, and therefore to be able to manage it. We measure to evaluate how close or far we are from the objectives we had set in our plan in terms of completion, quality, compliance to requirements, etc.
We measure also to be able to better estimate for new projects effort, cost and quality, based on past experience. Finally, we measure to evaluate how we improve on some key aspects of performance of the process over time, to see what are the effects of changes.
Measuring some key aspects of a project adds a non-negligible cost. So we do not measure just anything because we can. We must set very precise goals for this effort, and only collect metrics that will allow us to satisfy these goals.
There are two kinds of goals:
- Knowledge goals : they are expressed by the use of verbs like evaluate, predict, monitor. You want to better understand your development process. For example, you may want to assess product quality, obtain data to predict testing effort, monitor test coverage, or track requirements changes.
- Change or achievement goals : these are expressed by the use of verbs such as increase, reduce, improve, or achieve. You are usually interested in seeing how things change or improve over time, from an iteration to another, from a project to another.
Examples
- Monitor progress relative to plan
- Improve customer satisfaction
- Improve productivity
- Improve predictability
- Increase reuse
These general management goals do not translate readily into metrics. We have to translate them into some smaller subgoals (or action goals) which identify the actions project members have to take to achieve the goal. And we have to make sure that the people involved understand the benefits.
Examples
The goal to "improve customer satisfaction" would decompose into:
- Define customer satisfaction
- Measure customer satisfaction, over several releases
- Verify that satisfaction improves
The goal to "improve productivity" would decompose into:
- Measure effort
- Measure progress
- Calculate productivity over several iterations or projects.
- Compare the results
Then some of the subgoals (but not all) would require some metrics to be collected.
Example
"Measure customer satisfaction" can be derived from
- Customer survey (where customer would give marks for different aspects)
- Number and severity of calls to a customer support hotline.
For more information, consult [AMI95].
What is a metric?
We distinguish 2 kinds of metrics:
- A metric is a measurable attribute of an entity. For example, project effort is a measure (i.e., metric) of project size. To be able to calculate this metric you would need to sum all the time-sheet bookings for the project.
- A primitive metric is a raw data item that is used to calculate a metric. In the above example the time-sheet bookings are the primitive metrics. A primitive metric is typically a metric that exists in a database but is not interpreted in isolation.
Each metric is made up of one or more collected metrics. Consequentially each primitive metric has to be clearly identified and its collection procedure defined.
Metrics to support change or achievement goals are often "first-derivative" over time (or iterations or project). We are interested in a trend, not in the absolute value. To "improve quality" we need to check that the residual level of known defects diminishes over time.
Templates
Template for a metric
| Name | Name of the metric and any known synonyms |
| Definition | The attributes of the entities that are measured using this metric, how the metric is calculated, and which primitive metrics it is calculated from. |
| Goals | List of goals and questions related to this metric. Also some explanation as to why the metric is being collected. |
| Analysis procedure | How the metric is intended to be used. Preconditions for the interpretation of the metric (e.g., valid range of other metrics). Target values or trends. Models of analysis techniques and tools to be used. Implicit assumptions (e.g., of the environment or models). Calibration procedures. Storage. |
| Responsibilities | Who will collect and aggregate measurement data, prepare the reports and analyze the data. |
Template for a Primitive Metric
| Name | Name of the primitive metric |
| Definition | Unambiguous description of the metric in terms of the project’s environment |
| Collection procedure | Description of the collection procedure. Data collection tool and form to be used. Points in the lifecycle when data are collected. Verification procedure to be used. Where will the data be stored, format, precision. |
| Responsibilities | Who is responsible for collecting the data. Who is responsible for verifying the data. |
Metrics activities
There are 2 activities:
- Define measurement plan
- Collect measures
Define measurement plan is done once per development cycle - in the inception phase, as part of the general planning activity, or sometimes as part of the configuration of the process in the development case. The measurement plan may be revisited like any other section of the software development plan during the course of the project.
Collect measures is done repetitively, at least once per iteration, and sometimes more often (e.g., weekly on an iteration spanning many months).
The metrics collected are part of the Status Assessment document, to be exploited in assessing the progress and health of the project. They may also be accumulated for later use in project estimations and trends over the organization.
For the role of metrics in project management, read [ROY98].
