NEWS AT SEI
This article was originally published in News at SEI on: July 1, 2008
The CMMI Steering Group and the SEI have agreed that the material contained in Version 1.2 of the CMMI for Development (CMMI-DEV) model should be clarified to better explain high maturity. These clarifications will be designed to address the needs that CMMI users, appraisers, and instructors have expressed related to CMMI high maturity practices. This column describes the reasons and current planning for how high maturity will be clarified.
Part of the purpose of the Version 1.2 release was to strengthen the integrity of CMMI appraisal results. An important aspect of this purpose included high maturity and capability (i.e., levels 4 and 5). Actions taken as part of Version 1.2 included the following:
The SEI hired two experienced lead appraisers to augment its quality assurance efforts early in 2008 and began conducting dedicated high maturity audits in February. As of October 2008, the SEI has conducted 29 audits of high maturity appraisals. These audits have been conducted both onsite and remotely. The time differences involved in some appraisals conducted half-way round the world from the audit team adds to the challenges faced by both the appraisal team and the audit team.
Remote audits involve an audit team that requests and receives appraisal information on high maturity elements from the appraisal team as the appraisal is being conducted. These high maturity elements are strong indicators that the organizations (and the high maturity lead appraisers) fully understand the fundamentals of high maturity and are properly using the tools needed to achieve high maturity and high capability (i.e., statistical management and process improvement models).
Audits are difficult to conduct, but are worth the trouble because of the valuable information collected. The valuable information received from those audited has confirmed that these audits serve a useful purpose of maintaining the integrity of appraisal results. We will soon shift our focus from auditing all high maturity appraisals to a sampling approach. This approach will assure that we do not focus exclusively on high maturity. There is a much greater number of appraisals aimed at achieving maturity levels 2 or 3 and others that focus on achieving capability profiles, without seeking maturity level equivalence.
Usually, audits confirm the findings of the appraisal team. Such audit results not only confirm the appraisal results for the organization, but also confirm the process and judgment used by the lead appraiser and his or her team.
Occasionally, audits uncover appraisal deficiencies. Examples of deficiencies include errors in judgment by high maturity lead appraisers and the evidence provided by the organization being appraised. In such instances, audit teams work closely with lead appraiser to ensure that the appraisal is brought to an appropriate conclusion. The outcomes of audits in these situations have resulted in the decertification of some high maturity lead appraisers, and in some cases rejection of submitted appraisal results.
The CMMI Steering Group and the SEI have agreed that they want to provide more help for users targeting level 4 and 5 appraisal results. Although there are white papers now available on the SEI website that provide clarification of high maturity practices, all agree that more is needed.
The first initiative is to create a set of criteria that describes the particular elements of the appraisal that are being audited. We think that communication of these criteria can help alleviate fear of the unknown that all audits can cause. The initial criteria will be identified with leadership from industry and approved by the Steering Group to assure that the breadth of model interest and experience is represented. As with other CMMI elements, the SEI will provide a change request mechanism so that further improvements can be made.
Work is also being done to craft improvements to the text of the CMMI-DEV model. These improvements are being developed in the form of model redlines. These redlines were reviewed by the high maturity lead appraisers and the CMMI Steering Group at a workshop in late September. As a result of the review, the Steering Group determined that modernizing the practices for levels 4 and 5 was a better choice than a more limited attempt to clarify the information. These improvements will be incorporated into the planned release of CMMI, Version 1.3 for all three CMMI constellations rather than in a CMMI-DEV, Version 1.2a model.The Steering Group and the SEI also recognize that suggested improvements to high maturity practices must come from a wider community than only the appraisers. Therefore, more opportunities like the September workshop are being considered. Where possible, we will link with events such as the November CMMI Conference in Denver and the March SEPG Conference in San Jose by providing opportunities to contribute ideas in special sessions held at these conferences.
Also planned are improvements and clarifications to the auditing processes now conducted by the SEI to evaluate high maturity appraisals. If an audit results in decertification, an appeal process has been defined that will provide a way for the lead appraiser to request a review of the judgment made by the CMMI Appraisal Program. Documentation of the appeal process is being distributed to high maturity lead appraisers for their use. This appeal process includes a review of the contested appraisal by a board of lead appraisers, with the voting members from outside the SEI.
If the result of an audit is that the appraisal results are rejected by the SEI, a process is available to the organization to contest this outcome. For organizations concerned about appraisal results, an adjudication process is being made available to assure the organization’s understanding of the reasons for the SEI’s determination. Further, the process allows a review by high maturity lead appraisers external to the SEI to provide an independent cross-check of the SEI quality assurance determination. Detailed descriptions of these processes are found on the SEI's Appraiser Program pages. www.sei.cmu.edu/appraisal-program/appraiser-communications/index.html.
High maturity thinking has come a long way since the levels associated with the term were created in the early 1990s. The effective use of statistical techniques to manage and improve processes has advanced with the continuing evolution of tools that assist in measuring and predicting process performance. The CMMI Steering Group and SEI believe that the steps described in this column will help ensure that CMMI users more clearly understand and interpret CMMI practices and achieve their process improvement goals.
As the director of special projects at the Software Engineering Institute, Mike Phillips leads the Capability Maturity Model Integration (CMMI) project for the SEI. He was previously responsible for transition-enabling activities at the SEI. Prior to his retirement as a colonel from the Air Force, he managed the $36B development program for the B-2 in the B-2 SPO and commanded the 4950th Test Wing at Wright-Patterson AFB, Ohio. In addition to his bachelor’s degree in astronautical engineering from the U.S. Air Force Academy, Phillips has master’s degrees in nuclear engineering from Georgia Tech, in systems management from the University of Southern California, and in international affairs from Salve Regina College and the Naval War College.
The views expressed in this article are the author's only and do not represent directly or imply any official position or view of the Software Engineering Institute or Carnegie Mellon University. This article is intended to stimulate further discussion about this topic.
For more information
Please tell us what you
think with this short
(< 5 minute) survey.