Software Product Lines
Framework Home
Introduction
Product Line Essential Activities
Product Line Practice Areas
Software Engineering
Practice Areas
Technical Management
Practice Areas
Configuration Management
Make/Buy/Mine/Commission Analysis
Measurement and Tracking
Process Discipline
Scoping
Technical Planning
Technical Risk Management
Tool Support
Organizational Management
Practice Areas
Frequently Asked Questions
Glossary
Bibliography

A Framework for Software Product Line Practice, Version 5.0

Next Section Table of Contents Previous Section

Measurement and Tracking

Software development efforts must be tracked to ensure that they meet their organizational goals. While informal and qualitative tracking techniques are valuable, they should be supplemented with objective and quantitative measurement techniques.

The purpose of measurement is to support project tracking and guide management decision making [Grady 1992a, Park 1996a]. The manager sets goals, defines objectives that satisfy those goals, and then creates a plan and applies resources to achieve those objectives. In order to determine whether the goals are being achieved as time passes (that is, whether the plan is working), the manager must have data that indicate the state of the effort. By tracking and analyzing relevant measurable attributes of the effort's process and product, the manager has a quantitative window on the progress toward the effort's goals. The manager can also detect issues that indicate when the effort has diverged from expectations. The manager can then revise the effort's goals, plan, or resources to address these issues–or recalibrate everyone's expectations.

Measurement-based project tracking is based on

Once the plan is established, adherence to it should be monitored continually to ensure that the measures are being used effectively. The plan should also be revised as the organization's and effort's goals evolve.

Aspects Peculiar to Product Lines

The techniques for collecting and tracking data are the same for a product line as for a single system; both situations require an initiation phase to plan for the measurements and a performance phase to collect data and analyze results. However, in a product line, data collection must provide information from three perspectives, not just the single perspective of product development. Recall the three essential activities of product line development:

  1. core asset development, comprising the efforts needed to produce reusable assets and the supporting infrastructure for their use
  2. product development, comprising the efforts needed to produce individual products for customers
  3. management of the overall product line, including the strategic planning and direction of a total product line enterprise

Appropriate measures must be collected and tracked to support each of these activities. A product line manager and/or a product line management oversight group is concerned with tracking whether the overall multiproduct effort is efficient, effective, and progressing properly toward achieving its strategic goals and satisfying the product line's production constraints (see "Core Asset Development"). Managers of core asset development are concerned with the quality and usefulness of the core assets they produce and the productivity of the people who produce them. Individual product managers are concerned with the efficiency of their staff and the quality of their products.

These differing concerns are complementary in that the measures required to track the progress of the overall product line effort are mostly aggregated from the measures required to track the progress of its constituent core asset and product efforts. For example, the product line goal Better Quality (which might be indicated by fewer errors after delivery) across all products is tracked in terms of

The quality level of the individual products is influenced heavily by the quality of the core assets from which they are built. So having fewer errors in core assets after their delivery to the product developers becomes a local goal of core asset development and is also tracked.

As another example, consider the product line goal Increased Profitability of Product Development. The profitability is tracked by the measures reported while component core assets are assembled according to the production plan. The cost measures associated with the development and evolution of the core assets also need to be factored into the profitability measurement.

Over and above the specifics of which measures to track and which data to collect for software product lines, this practice area has another dimension that makes it highly relevant. The ability of an organization to collect and analyze data and to track measures about itself at all says a lot about the organization: it is comfortable with disciplined processes, accustomed to taking a long-term view, and interested in self-improvement–all of which are the hallmarks of successful product line organizations.

Application to Core Asset Development

Management overseeing core asset development has two concerns: (1) the efficiency of the core asset effort and (2) its effectiveness in benefiting the associated product efforts that are its clients.

To meet efficiency goals, management should focus on tracking the cost and time required to develop the core assets. Satisfying efficiency goals means that core assets will have a minimum investment cost (therefore, requiring a lower payback from product efforts) and be available sooner for product efforts to use. These benefits mean that the overall product line effort will spend less on both core assets and products, permitting it to either lower prices or increase profits as the company's business strategy and market conditions dictate.

To meet effectiveness goals, the manager should focus on providing the core assets that offer the best chance of reducing the amount work needed in product efforts. The table below tells what such effectiveness measures can indicate and help determine.

Effectiveness Measures

Measures for effectiveness indicate

Which helps to determine

which core assets are used by product efforts and how often

whether the available core assets are useful

how many bugs are found in core assets by the product developers

the quality of the core assets

how much product efforts expend in finding, tailoring, and integrating assets

needed improvements in supporting infrastructure

where product efforts spend time otherwise

opportunities for future core asset or infrastructure work

Application to Product Development

The measurement activity for product development efforts has two facets: one that's conventional and one that's product line specific. The conventional facet involves the collection and analysis of measures needed for the management of any product development effort. For example, managers of product development efforts in general need to track the time and cost of their work activities, the quality of the products developed, and customer satisfaction. These particular measures are applicable to both products developed from core assets and those that are not.

The facet specific to product lines involves the collection of measures that product line and core asset development managers need in order to be effective managers, but those measures can only come from product development. The ones needed for the management of core asset development relate to the quality and usability of core assets from the perspective of the product efforts. Included are measures such as which core assets were used in which products, the amount of effort required to incorporate a core asset into a product, and the number of problems encountered. The measures needed for product line management are generally a subset of those needed to manage core asset and product efforts, but they are aggregated across all the product line's efforts. Product line management will use these measures to determine whether resources are allocated properly between core asset and product efforts and whether these efforts are achieving the intended market results for the business.

Example Practices

Goal-driven software measurement (GDSM): GDSM [Park 1996a] is a ten-step technique that supports project tracking by defining the appropriate project measures based on the project goals and developing a plan to operationalize and verify those measures. GDSM is based on the earlier work of Basili and Weiss [Basili 1984a]. The ten steps of GSDM are

  1. Define the goals.
  2. Refine the goals using clarifying questions.
  3. Define subgoals for the relevant stakeholders.
  4. Operationalize the goal statements.1
  5. Determine the success criteria for the stakeholder.
  6. Determine the success, progress, and analysis indicators associated with those criteria.
  7. Define the organizational strategies and activities for achieving their goals.
  8. Determine the measures and data elements needed by all identified indicators.
  9. Assess the current infrastructure and identify actions needed to implement the measures.
  10. Develop a measurement plan that addresses the actions to be taken and how the measurements will be verified.

Choosing measures: A good starting point for choosing product line measures comes from the work of Zubrow and Chastek [Zubrow 2003a]. They describe indicators that are of interest to a product line manager, a core asset development manager, and a product development manager (see the table below). Under each category, a broad set of measures (listed in the "Goal" column of the table) is defined that returns information about performance (measuring cost, schedule, and quality of product efforts), compliance (measuring the adherence of the product line effort to established procedures and processes), and effectiveness (characterizing how the overall product line effort is meeting its goals).

Product Line Indicators and Measures [Zubrow 2003a]

Goal

Product Line Manager

Core Asset Development Manager

Product Development Manager

Improved Performance

  • total product development cost
  • productivity
  • schedule deviation
  • time to market
  • trends in defect density
  • number of products (past, current, future)
  • time spent on life-cycle activities
  • cost to produce core assets
  • cost to produce infrastructure
  • schedule deviation
  • defect density in core assets
  • number and type of artifacts in asset library
  • direct product cost
  • defect density in application artifacts
  • percent reuse

Compliance

  • mission focus
  • process compliance
  • mission focus
  • process compliance
  • process compliance

Increased Effectiveness

  • return on investment
  • market satisfaction
  • core assets utility
  • core assets cost of use
  • percent reuse
  • customer satisfaction


"Building a Business Case: Example Practices" discusses specific measures for supporting a business case.

Collecting data: Different types of measures require different data collection techniques. Common techniques include

Each of these techniques can be applied manually or with the assistance of automated tools, depending on the size of the effort being undertaken and whether time or money is the more limited resource.

Reuse measures: A higher level of software reuse is not, in itself, an end goal of a product line effort but merely a strategy for achieving goals such as shorter time to market. However, because software reuse is such a cornerstone of the product line strategy, it is a useful quantity to measure. Beyond that, it is useful to have a measure-based model of what the reuse is buying you, in terms of cost avoidance, return on investment, or some other goal-related quantity. Poulin provides several such models [Poulin 1997a]. Some of the proposed models also have associated tools that support the appropriate analyses and presentation of results.

Practice Risks

A poor data collection and measurement program will be a waste of time and resources, will have an opportunity cost (i.e., a loss in terms of the productive work the staff could have been doing instead), and will cause resentment and mistrust of future measurement efforts. It will also fail to inform management of where the organization stands with respect to meeting its product line goals. This failure can, in turn, lead to uninformed management decisions that can undermine the product line efforts and ultimately to an inability to validate the business case that justified that product line effort. Potential causes of inadequate data collection and measures are as follows:

Further Reading

[Geppert 2003a]
Geppert and Weiss describe Avaya Labs' method for the quantitative assessment of potential product line domains. Their paper defines measures for the potential corporate impact and the likelihood of success when a candidate domain is implemented as a product line.

[Park 1996a]
Park, Goethert, and Florac provide an extensive guide to establishing a measurement activity based on business goals.

[Poulin 1997a]
Poulin treats measurement from the point of view of a reuse organization. His work describes a good suite of reuse-based measures and models.

[Zubrow 2003a]
Zubrow and Chastek suggest a small set of measures to track a software product line. Those measures range from the relatively mature to the not yet validated.

Next Section Table of Contents Previous Section

 

1 This step involves determining what properties and attributes will be analyzed and why, the environment and context of the collected data, who will use that data, and who will use the results.