Digital Intelligence & Forensics
Measurement & Analysis
Performance & Dependability
Process & Performance Improvement
Security & Survivability
NEWS AT SEI
This article was originally published in News at SEI on: January 1, 2009
High maturity processes and their effects on performance have long been seen as valuable to organizations’ process improvement. This column moves beyond that notion to describe how process improvement in an organization can benefit the organization’s customers.
From the inception of the first Capability Maturity Model (CMM) in the late 1980s, the Department of Defense (DoD), the primary sponsor of SEI work, sought ways to gain confidence in the software development processes used by its suppliers. The now widely known five-level maturity rating scale was created. It identified reasonable groups of process improvement activities that could be used to improve organizations’ process discipline and quality focus.
Performance improvements achieved as a result of this process improvement have been documented in case studies such as those at the SEI website and now at www.thedacs.com (you must be a member of the DACS to login and browse this information).
Some acquisition organizations, both in industry and government, have used maturity levels in the selection of their suppliers. Some use a maturity-level rating in requests for proposals (RFPs) as entry criteria for accepting contract proposals at the beginning of the selection process. Others consider maturity-level ratings as a factor in rating suppliers as “acceptable” or “preferred.” Maturity Level 3 has been the most commonly used—and abused—level rating.
Because appraisal results are summarized into a single maturity-level rating number, these ratings can often mislead acquiring organizations that are using maturity levels to evaluate potential suppliers. To address this potential problem, a portion of the CMMI Product Team, consisting of members from government and industry, developed a guidebook to help acquisition organizations to more accurately interpret maturity-level ratings. The guidebook helps customers (i.e., acquirers) to collect better information and to gauge and manage the risks encountered when developing the complex and unprecedented systems sought in today’s RFPs. This guidebook was published as the SEI technical report, Understanding and Leveraging a Supplier’s CMMI Efforts: A Guidebook for Acquirers.
As organizations began to achieve high maturity status, expectations began to expand about what high maturity meant. Process improvement in an acquisition or development environment is fundamentally intended to result in products and services with higher quality and fewer defects. Process improvement should lead to performance improvement on real projects that deliver real software-intensive systems, particularly those sought by the DoD. Consequently, both customers and suppliers began to link high maturity processes to better performance expectations.
Unfortunately, a study conducted by the Defense Contract Management Agency (DCMA) across a large number of DoD programs found that there were “performance shortfalls” in many projects within companies that earned high maturity ratings. Most of these performance shortfalls were more closely linked to cost and schedule performance than to system performance or quality of products and services.
Research suggests that there are multiple causes of cost and schedule problems that extend far beyond the process dimension. Causes most often include requirements changes or other dynamics that process discipline cannot overcome. However, such issues and limitations of process improvement make it particularly useful to describe high maturity benefits for both the improving organization and its customers.
If a DoD acquirer chooses to be a customer of a high maturity company, what kinds of performance should they expect to see? What characteristics change as organizations move from level to level?Figure 1 is helpful in illustrating performance changes across increasing maturity levels.
As organizations move from the ad hoc approach of Maturity Level 1 up to Maturity Level 2, more accurate estimates of time and resources begin to occur, replacing the over-optimistic approaches used before. Historical data begins to influence plans. Notice that more accurate estimates are often of longer duration or higher cost than the estimates delivered at Maturity Level 1. At Maturity Level 3, the organization sees results of qualitative process improvement (i.e., using best practices in Organizational Process Focus) across its projects and reduction in the average duration and cost estimates that rose in Maturity Level 2.
At Maturity Levels 4 and 5, quantitative improvement is the dominant ingredient, and the reduction in the average time and resource commitment continues. But notice also that the tail of the curve remains. High maturity is not a guarantee of performance—but it increases the probability of high performance. Put another way, the overall program risk, particularly on the typical DoD program seeking to create unprecedented systems, is likely to be reduced if a high maturity provider is chosen.
What else do we need to consider? Another element of performance is the role of the program team. We often hear anecdotal evidence that a high maturity prime contractor may be viewed as performing below expectations when less mature teammates are the source of difficulties. As we have described in earlier columns, the value of CMMI for Acquisition and the contribution of the acquirer (i.e., customer) cannot be underestimated. Often, the impact of high maturity is diluted by other contributing elements in the equation. The acquirer must be aware of issues such as contractual requirements that maintain a specific agreed-on set of processes. This awareness may have the project feeding potential process-improvement ideas back to the customer organization, but limiting the ability of the project to significantly change.
What should a customer expect? An excellent presentation that provides answers to this question as observed within an organization with many high maturity projects was presented by Rick Hefner of Northrop-Grumman at the 2005 Annual Systems and Software Technology Conference (SSTC). You can see this presentation on the SEI website at http://www.sei.cmu.edu/cmmi/adoption/pdf/05SSTC-Hefner-Customer-Benefits-031905.pdf. For this column, I’ll elaborate on one of the slides in his presentation (Figure 2), highlighting the essential ingredients of the four process areas that are relevant.
When using OPP practices, there is an opportunity to create a collection of processes and sub-processes with valuable quantitative documentation. (This documentation is accumulated as the organization applies its measurement and analysis techniques to the activities seen across the organization’s projects.) This quantitative documentation is a source of the quantitative performance baselines and performance models needed to predict the effect of making sensible changes to controllable variables (e.g., personnel and engineering tools). Because an element of OPP describes a link between quality goals and objectives and actual process performance, the customer may be able to participate in the establishment of goals and objectives that may be mutually important to the company and its customer.
Now the projects have to perform and use the organizational assets to aid in rolling out the new or revised development processes. The first goal focuses on the quantitative elements, while the second goal dives deeper into the use of good statistical analysis to assist in process analysis and improvement. Often the customer can encourage improvement by seeking to understand some of the special causes of variation that the project is encountering, or perhaps those faced by previous projects that are now risks to the new project. It is not uncommon for some of the causes of variation to be addressed by a cooperative acquirer/supplier team.
Using this Maturity Level 5 based process area, those working on the project can research ways to deal with process problems they confront. In one project, the combination of COTS hardware with the project’s software packages using existing processes was causing delays. A CAR event analyzed the various subprocesses involved in successful integration and provided a revision that satisfied both the project and the customer. (The event had customer involvement, and some of the activities actually took place at the customer site.)
This process area adds the final elements to the high maturity suite, because it allows full implementation of quantitatively improved processes. One of the OID practices expects the organization to pilot changes and gather data on the pilot. Here again the customer can be part of the improvement effort in many cases.
The move to high maturity by an organization is typically seen as valuable for understanding its processes better and effecting change that reduces time to market and reduces the demand for resources that can be applied to grow the business. However, the organization’s customer can benefit as well. I want to leave you with the idea that the most significant value of high maturity is in the reduction of risk. The least certain value of high maturity is a guaranteed performance.
As the director of special projects at the Software Engineering Institute, Mike Phillips leads the Capability Maturity Model Integration (CMMI) project for the SEI. He was previously responsible for transition-enabling activities at the SEI. Prior to his retirement as a colonel from the Air Force, he managed the $36B development program for the B-2 in the B-2 SPO and commanded the 4950th Test Wing at Wright-Patterson AFB, Ohio. In addition to his bachelor’s degree in astronautical engineering from the U.S. Air Force Academy, Phillips has master’s degrees in nuclear engineering from Georgia Tech, in systems management from the University of Southern California, and in international affairs from Salve Regina College and the Naval War College.
For more information
Please tell us what you
think with this short
(< 5 minute) survey.