NEWS AT SEI
This article was originally published in News at SEI on: July 1, 2008
The CMMI Steering Group and the SEI have agreed that the material
contained in Version 1.2 of the CMMI for Development (CMMI-DEV) model
should be clarified to better explain high maturity. These
clarifications will be designed to address the needs that CMMI users,
appraisers, and instructors have expressed related to CMMI high
maturity practices. This column describes the reasons and current
planning for how high maturity will be clarified.
What will be clarified?
Part of the purpose of the Version 1.2 release was to strengthen the
integrity of CMMI appraisal results. An important aspect of this
purpose included high maturity and capability (i.e., levels 4 and 5).
Actions taken as part of Version 1.2 included the following:
- A certification process for high maturity lead appraisers was established.
- An appraisal lifetime policy was established, which limited the viability of appraisal results to three years.
- Rigorous reviews of appraisal results were defined and required to enable public announcements of appraisal rating results.
- Resources were committed to allow audits of all appraisals
targeting level 4 and 5 appraisal ratings for several months. These
audits have helped us assure that the fundamentals for CMMI high
maturity are in evidence in the appraised organizations claiming to
achieve these levels. More information about the policy regarding high
maturity appraisal audits can be found at www.sei.cmu.edu/appraisal-program/appraiser-communications/index.html.
The SEI hired two experienced lead appraisers to augment its quality
assurance efforts early in 2008 and began conducting dedicated high
maturity audits in February. As of October 2008, the SEI has conducted
29 audits of high maturity appraisals. These audits have been conducted
both onsite and remotely. The time differences involved in some
appraisals conducted half-way round the world from the audit team adds
to the challenges faced by both the appraisal team and the audit team.
Remote audits involve an audit team that requests and receives
appraisal information on high maturity elements from the appraisal team
as the appraisal is being conducted. These high maturity elements are
strong indicators that the organizations (and the high maturity lead
appraisers) fully understand the fundamentals of high maturity and are
properly using the tools needed to achieve high maturity and high
capability (i.e., statistical management and process improvement
Audits are difficult to conduct, but are worth the trouble because
of the valuable information collected. The valuable information
received from those audited has confirmed that these audits serve a
useful purpose of maintaining the integrity of appraisal results. We
will soon shift our focus from auditing all high maturity appraisals to
a sampling approach. This approach will assure that we do not focus
exclusively on high maturity. There is a much greater number of
appraisals aimed at achieving maturity levels 2 or 3 and others that
focus on achieving capability profiles, without seeking maturity level
Usually, audits confirm the findings of the appraisal team. Such
audit results not only confirm the appraisal results for the
organization, but also confirm the process and judgment used by the
lead appraiser and his or her team.
Occasionally, audits uncover appraisal deficiencies. Examples of
deficiencies include errors in judgment by high maturity lead
appraisers and the evidence provided by the organization being
appraised. In such instances, audit teams work closely with lead
appraiser to ensure that the appraisal is brought to an appropriate
conclusion. The outcomes of audits in these situations have resulted in
the decertification of some high maturity lead appraisers, and in some
cases rejection of submitted appraisal results.
So what is the plan?
The CMMI Steering Group and the SEI have agreed that they want to
provide more help for users targeting level 4 and 5 appraisal results.
Although there are white papers now available on the SEI website that
provide clarification of high maturity practices, all agree that more
The first initiative is to create a set of criteria that describes
the particular elements of the appraisal that are being audited. We
think that communication of these criteria can help alleviate fear of
the unknown that all audits can cause. The initial criteria will be
identified with leadership from industry and approved by the Steering
Group to assure that the breadth of model interest and experience is
represented. As with other CMMI elements, the SEI will provide a change
request mechanism so that further improvements can be made.
Work is also being done to craft improvements to the text of the
CMMI-DEV model. These improvements are being developed in the form of
model redlines. These redlines were reviewed by the high maturity lead
appraisers and the CMMI Steering Group at a workshop in late September.
As a result of the review, the Steering Group determined that
modernizing the practices for levels 4 and 5 was a better choice than a
more limited attempt to clarify the information. These improvements
will be incorporated into the planned release of CMMI, Version 1.3 for
all three CMMI constellations rather than in a CMMI-DEV, Version 1.2a
model.The Steering Group and the SEI also recognize that suggested
improvements to high maturity practices must come from a wider
community than only the appraisers. Therefore, more opportunities like
the September workshop are being considered. Where possible, we will
link with events such as the November CMMI Conference in Denver and the
March SEPG Conference in San Jose by providing opportunities to
contribute ideas in special sessions held at these conferences.
Also planned are improvements and clarifications to the auditing
processes now conducted by the SEI to evaluate high maturity
appraisals. If an audit results in decertification, an appeal process
has been defined that will provide a way for the lead appraiser to
request a review of the judgment made by the CMMI Appraisal Program.
Documentation of the appeal process is being distributed to high
maturity lead appraisers for their use. This appeal process includes a
review of the contested appraisal by a board of lead appraisers, with
the voting members from outside the SEI.
If the result of an audit is that the appraisal results are rejected
by the SEI, a process is available to the organization to contest this
outcome. For organizations concerned about appraisal results, an
adjudication process is being made available to assure the
organization’s understanding of the reasons for the SEI’s
determination. Further, the process allows a review by high maturity
lead appraisers external to the SEI to provide an independent
cross-check of the SEI quality assurance determination. Detailed
descriptions of these processes are found on the SEI's Appraiser Program pages. www.sei.cmu.edu/appraisal-program/appraiser-communications/index.html.
High maturity thinking has come a long way since the levels
associated with the term were created in the early 1990s. The effective
use of statistical techniques to manage and improve processes has
advanced with the continuing evolution of tools that assist in
measuring and predicting process performance. The CMMI Steering Group
and SEI believe that the steps described in this column will help
ensure that CMMI users more clearly understand and interpret CMMI
practices and achieve their process improvement goals.
About the Author
As the director of special projects at the Software Engineering
Institute, Mike Phillips leads the Capability Maturity Model Integration (CMMI) project for the SEI. He was previously responsible for transition-enabling
activities at the SEI. Prior to his retirement as a colonel from the Air
Force, he managed the $36B development program for the B-2 in the B-2 SPO
and commanded the 4950th Test Wing at Wright-Patterson AFB, Ohio. In addition
to his bachelor’s degree in astronautical engineering from the U.S.
Air Force Academy, Phillips has master’s degrees in nuclear engineering
from Georgia Tech, in systems management from the University of Southern
California, and in international affairs from Salve Regina College and the
Naval War College.
views expressed in
this article are the
author's only and do
not represent directly
or imply any official
position or view of
the Software Engineering
Institute or Carnegie
This article is intended
to stimulate further
discussion about this