features
First Workshop on CMMI High Maturity Measurement Brings Organizations Together
Organizations are increasingly looking for guidance on what it takes to reach high maturity and how to keep improving once they get there. As high-maturity organizations work to improve their use of measurement and analysis, they often look to examples of successful implementations for direction. In response to the need for clarification and guidance on implementing measurement and analysis in the context of high-maturity processes, members of the SEI’s Software Engineering Measurement and Analysis (SEMA) initiative organized a workshop at the 2008 SEPG North America conference to bring leaders in the field together at a forum focused specifically on this topic.
Organizations were invited to share information about two important aspects of their measurement and analysis practice: process performance baselining and modeling. Workshop participants were from organizations using process-performance models and baselines. Several representatives from Hill Air Logistics Center, Lockheed-Martin, Northrop Grumman, and Raytheon attended the two-day workshop. “This workshop provided us with a great opportunity to both learn from each other through the sharing of best practices and further build upon our network of fellow practitioners for future collaboration,” says Neal Mackertich, a participant from Raytheon.
Workshop goals
The main goals of the workshop were to
- allow CMMI high maturity organizations to share best practices and case studies
- identify ways to develop CMMI high maturity measurement and analysis practices and accelerate their adoption
- enable networking among practitioners
Organizations gave 20-minute presentations summarizing their past experiences and future plans related to the following topics:
- barriers faced by their organizations
- lessons learned in the deployment, training, adoption, and institutionalization of CMMI process performance baselines and models
- best practices and examples of valid, practical methods for implementing process performance models and baselines
- data quality and integrity issues
- plans for modeling over the next three to six months, including the nature of the performance outcomes and drivers most likely to be investigated
- suggestions for subject matter to include in future SEI state-of-the-practice surveys
A sampling of the discussion in each topic is presented in the following section.
Barriers faced
The organizations noted that they face challenges
- establishing the value of developing and using process-performance models (PPMs) and baselines (PPBs)
- convincing project managers to collect new measures to be used for their PPMs and PPBs
- retaining consistent operational definitions as the scope of their measurement and analysis activities expand
Lessons learned
Among the lessons learned, the organizations noted that
- useful PPMs and PPBs require domain and statistical knowledge. Neither alone is sufficient.
- coaching and mentoring are critical elements of the adoption strategy when developing and using the PPMs. This includes guidance on possible decisions and actions associated with results from PPMs and PPBs.
Best practices
Participants discussed what they considered to be best practices and tips for successful implementation of performance models and baselines, including
- providing education and tools to support modeling and analysis
- verifying data integrity before using the data for PPBs and PPMs
- performing product simulation and analysis in addition to process simulation and analysis
Data issues
Data quality and integrity problems that are encountered at lower maturity levels continue to threaten the potential value from PPMs and PPBs. Some examples include
- inconsistent operational definitions, which wreak havoc on modeling attempts
- manual data collection subject to human error
- problems aggregating and disaggregating data
- missing context information to go with the collected data
- consistency problems arising from decentralized databases
Modeling plans
Participating organizations plan to use the following modeling techniques in the next three to six months:
- Bayesian methods to calculate control limits during statistical management
- regression analysis to model and predict customer satisfaction
- measurement system evaluations to identify the degree of noise in data due to the measurement process
- discrete event simulation to facilitate Lean Six Sigma improvements
Subject matter for state-of-the-practice surveys
Workshop participants suggested the following topics of interest for future SEI state-of-the-measurement-practice studies:
- adoption and use of measurement and analysis related to high maturity practices, particularly the use of PPMs and PPBs
- balancing statistical and domain expertise to develop and sustain the value and use of PPMs and PPBs
- bases for choosing critical subprocesses to place under statistical control and ways in which to develop a collection of PPMs
- data archeology (i.e., creating baselines from paper records for previously unmeasured attributes)
- data quality and integrity
Future plans
The discussion during the workshop will be documented in greater detail in an SEI technical note.
The SEI plans to hold high maturity measurement workshops semi-annually to allow invited attendees to continue sharing their experiences and lessons learned in the adoption, development, and use of measurement and analysis in high maturity settings.
At the next workshop later this year, participants from the first workshop and other invited representatives from CMMI high maturity organizations will present their measurement and analysis procedures and initial results. SEI experts will offer additional guidance on high maturity topics and present pertinent results from the 2008 SEI State of the Measurement Practice survey.
Work products resulting from the next workshop will include
- thorough case studies of process performance models and their outcomes in high maturity organizations
- requirements definitions for a possible SEI course on the coaching, adoption, and institutionalization of CMMI process performance models and baselines
- plans for a coordinated empirical study of common performance outcomes and associated controllable and uncontrollable drivers of those outcomes
Subsequent workshops will be open to a larger group of CMMI high-maturity organizations. Organizations wishing to participle in future workshops must be willing to document and share their experiences with the use of measurement and analysis methods in relation to high-maturity practice. To ensure high-value workshops, the SEMA team will screen submissions prior to accepting an organization’s request to participate.
For more information about ULS systems, please contact—
Customer Relations
Software Engineering Institute
customer-relations@sei.cmu.edu