NEWS AT SEI

Do you play golf, jog, go bowling, ride a bicycle, lift weights, or play basketball? If you do, then you likely keep track of your performance. Perhaps it is as simple as, “I knocked three strokes off my game today,” or “I lifted 10 more pounds than I could last week.” People keep score like this because most are performance or achievement driven. They want to know how well they are doing—whether their performance is improving or declining—and how their performance compares with their own personal best or with the performance of others.

In much the same way, companies, organizations, and software projects attempt to understand their overall performance, compare it to others, and find ways to become better. When performance measurement is used for comparison, the measures to be compared must be commonly defined.

This is the major obstacle that has hampered effective software project performance comparison and benchmarking. Broadly speaking, measurement definition is not standardized in software development. Not only is it difficult or impossible to compare measures between projects from different organizations, but comparison within the same organization is fraught with error because different operational definitions are used for the same measure.

The SEI’s Software Engineering Process Management (SEPM) program launched a collaborative research effort to investigate ways to improve the practice of software project performance measurement. The collaboration brought together experts from some of the world’s top organizations that were already working in or had a strong interest in software project performance measurement. Table 1 lists the organizations that participated in this initiative.

Table 1: Collaborators

David Consulting Group Galorath Incorporated ISBSG Lockheed Martin Motorola Oracle PRICE Systems PRTM QSM Raytheon SEI SPR 4SUM Partners University of Ottawa

Through a series of workshops and group work sessions, a set of software project performance measures and influence factors was identified and defined.

The group started with a large list of candidate measures and factors, then used multi-voting to whittle the list down to a manageable set of terms. The following questions guided the selection process:

1. What are the key measures that best characterize project performance?
2. What factors impact performance in a significant way?
3. What measures and factors would be most useful for software project performance comparison?
4. How difficult would it be to collect the factor or measure?
5. Is the factor or measure currently being collected as part of your organization’s data repository?

The final list of performance measures is presented in Table 2, and the list of influence factors is presented in Table 3.

The newly published SEI technical report titled A Data Specification for Software Project Performance Measures: Results of a Collaboration on Performance Measurement(CMU/SEI-2008-TR-012) contains the full definitions for these terms.

Table 2: Performance Measures	Table 3: Influence Factors
Project effort	Size
Productivity	Artifact reuse
Project duration	Project type
Schedule predictability	Application domain
Requirements completion ratio	Average team size
Post-release defect density	Maximum team size
	Team expertise
	Process maturity
	Functional requirements stability

Performance measures focus on results, such as how long it took to run a certain distance. They answer the question, “What does success really mean?”

Influence factors are aspects of the environment that can impact the outcome of a performance measure. For example, if you want to compare how fast you run against others, you have to agree about the conditions under which the measurements will occur, such as terrain (flat vs. hilly), temperature (cold vs. hot), and distance (100 yards vs. 2 miles). In the context of running, these variables—terrain, temperature, and distance—are all influence factors. In software development, some influence factors are controllable by management, while others are not. When making comparisons between software projects, influence factors can be used to facilitate the comparison of projects that are similar to each other (with respect to one or more influence factors).

With data available from multiple projects that possess similar characteristics, a project’s performance can be compared against that of other projects to determine areas of strength and weakness. When used in this way, measurement comparison motivates process improvement.

Organizations just starting a measurement program do not have historical data on which to base their estimates, so they want to know what measures to use and what reasonable targets for their measures are. A Data Specification for Software Project Performance Measures is a good starting point for these organizations because it lists the key measures that every organization should collect.

Organizations that are more experienced in measurement want to compare their performance with competitors in their industry. If all organizations use the definitions in the specification document, comparison between companies will be more accurate and useful.

Finally, progressive organizations want to learn about the best practices used by industry leaders so they can adapt them for their own use through benchmarking.

In each of the above cases, the valid comparison of measurement data is an integral step in realizing these objectives. The use of common definitions for influence factors and performance measures is a prerequisite for valid comparison.

A more powerful use of performance measurement is within the context of benchmarking. Benchmarking is a process that uses performance measurement to identify best-in-class achievement (i.e., the benchmark), but goes beyond mere comparison to determine how the best-in-class achievement was attained. Once the how is understood, the enablers (e.g., methods, procedures, tools) that led to the stellar performance are adapted by an organization or project to improve and thereby achieve similar stellar performance.

The SEI is interested in feedback from and collaboration with organizations that intend to implement or are implementing the software project performance measures and influence factors specified in this document. If you would like to provide feedback or discuss collaboration, contact us using the links in the For More Information box at the bottom of this page.