Myths about SCAMPI Appraisals

NEWS AT SEI

Author

Mike Phillips

This library item is related to the following area(s) of work:

Process Improvement
CMMI

This article was originally published in News at SEI on: January 1, 2004

At the CMMI Technology Conference this past November, Dave Kitson, the SEI’s chief architect of the SCAMPI V1.1 Method, presented “Ten Myths about SCAMPI.” (He started with 10, but by the time he actually made his presentation, there were 14….) With minor elaboration and Dave’s kind permission, I’m including them in this column, as the concerns they raise continue to be heard.

The myths are presented in no particular order, and were “discovered” without any particular scientific or statistical methodology. The myths originated from one or more of the following:

  • SEI observations of SCAMPIs
  • inquiries made to the SEI (phone, email)
  • questions from students attending SEI classes (Intro to CMMI, Intermediate CMMI, SCAMPI Lead Appraiser Training)

Myth 1
Findings are required to be reported at the sub-practice level.

One of the difficult construction decisions in developing the CMMI was determining what level of inquiry was appropriate for CMMI guidance in appraisals. We determined that while goal satisfaction was “rated,” (all goals in a process area [PA] must be “satisfied” in order for the PA to be “fully satisfied” for maturity level or capability level ratings) specific and generic practices (the next level down in the models) would be examined to check if they were being implemented—but subpractices would not be. However, findings may be reported at the sub-practice level if requested by the appraisal sponsor.

Typically, however, findings are reported as goal-level statements that summarize gaps in practice implementation. These statements must be abstracted to the level of the organizational unit, and cannot focus on individual projects (unless the option for project-specific findings has been agreed upon during planning).

Myth 2
The Appraisal Disclosure Statement (ADS) is optional.

The ADS is a required part of the SCAMPI method; it is typically prepared by the SCAMPI lead appraiser and must be provided to the appraisal sponsor as well as the SEI.

The SCAMPI lead appraiser affirms that the information in the ADS is accurate and that the appraisal was conducted in full accordance with the requirements of the SCAMPI V1.1 Appraisal Method and the provisions of his or her authorization as a SCAMPI V1.1 lead appraiser. The ADS is intended to serve as a common basis for the disclosure of SCAMPI appraisal results, and is now the basis for any SEI cataloging of appraisal results that the sponsor agrees to make publicly available.

Myth 3
SCAMPI appraisal team members must be trained in CMMI within 60 days of the appraisal onsite.

SCAMPI appraisal team members must receive Introduction to CMMI model training from an SEI-authorized instructor at any time prior to participation as a team member. (This can be provided by the lead appraiser if—and only if—the individual has also qualified as an instructor.) Appraisal team training is typically provided by the lead appraiser, except in some large organizations that provide organizational team training for their appraisal team members. There are currently no requirements for renewal or refresher training.

Myth 4
SCAMPI cannot be used in discovery mode.

Discovery mode, in which little or none of the needed objective evidence is readily accessible by the appraisal team, is an acceptable mode of use for SCAMPI. However, this mode of use is the most expensive and time consuming. 

Further, excessive or repeated employment of this mode of use may suggest an organization that is not managing process improvement in a way that provides sufficient visibility to management. The whole idea of practice implementation indicators is that these constitute a valuable resource for the organization itself, whether it ever conducts a formal appraisal or not.

Myth 5
Having two direct artifacts obviates the need for indirect or affirmation objective evidence.

This myth is a result of misunderstanding the significance of a direct artifact as well as the purpose of the requirement for indirect or affirmation objective evidence. The number of direct artifacts pertinent to a practice depends on the practice as well as its implementation in an organization. One direct artifact might show software planning, for example, while another might represent the systems engineering master plan.

The requirement for indirect or affirmation objective evidence meets the Appraisal Requirements for CMMI (ARC) requirement for corroborating objective evidence. These assure that the direct artifacts were created within an understood process, or are being used and maintained.

Myth 6
Alternative practices must be one-for-one replacements for model practices.

In the CMMI Product Suite the term “alternative practice” is defined as “a practice that is a substitute for one or more generic or specific practices contained in CMMI models that achieves an equivalent effect toward satisfying the generic or specific goal associated with model practices. Alternative practices are not necessarily one-for-one replacements for the generic or specific practices.”

Thus, it may be that an organization is able to achieve a goal with a different number of practices than is expected by the CMMI models.

Myth 7
If we conduct a SCAMPI appraisal using the staged representation, we must report a maturity level.

The SCAMPI method requires that, as a minimum, goal ratings be performed; there is no requirement that any further rating activities be performed. Furthermore, even if additional ratings are rendered by the appraisal team, the appraisal sponsor can decide how to communicate these ratings within the organization.

Myth 8
If we use the staged representation, we must examine all process areas up to and at the target maturity level.

The model scope of a SCAMPI appraisal is governed by the appraisal outputs selected by the appraisal sponsor and the inherent relationships among CMMI process areas. If a maturity level rating is not selected as an appraisal output, the model scope could be as small as one process area, irrespective of the representation chosen.

Myth 9
Appraisal team members retain full productivity until 4:00 am.

No one’s interests are served by agreeing to unrealistic appraisal plans. The SCAMPI lead appraiser has primary responsibility for ensuring that the appraisal plan is realistic. The SCAMPI documentation does not “require” the appraisal to be finished in a specific amount of time, so it may be necessary to change the plan if excessive hours are being spent on the appraisal.

Myth 10 (1/2)
If all of the goals are satisfied during an ARC class B, we should be able to render a maturity level rating.

First, ARC class B appraisals do not produce goal ratings; that is, they do not sanction goal ratings as an output of a class B appraisal. Ratings are reserved for ARC class A appraisals, and the CMMI community has invested significantly in a standard appraisal method for this purpose—the “Class A” SCAMPI.

Second, class B appraisal methods are exempt from some key ARC requirements pertaining to data consolidation and validation, thus providing less rigor than an ARC class A method in the accuracy of the resulting findings. Thus, the CMMI Product Development Team decided that the rendering of ratings is not warranted for class B or C methods.

Finally, the team believed that allowing appraisal methods with significantly varying degrees of rigor to produce ratings would eventually result in an erosion of confidence and support for ARC class B and C methods, thus depriving these classes of appraisal methods of their useful role in process improvement.

Many organizations execute high-fidelity class B appraisals that assure readiness for class A. If they have not been planned and executed as a class A, however, they simply cannot count for the purpose of rendering a maturity level.

Myth 11
There are only 10 SCAMPI myths.

Just a little humor!

Myth 12
Periodic progress reviews are mandatory for maintaining a maturity level rating.

A variation of this myth is that a SCAMPI appraisal must be repeated every two years to be “valid.” At this time, there is no SEI position on duration for SCAMPI appraisal outputs.

While there are currently no formal requirements for periodic progress or status reviews (or appraisals), such reviews are a common practice in related methodologies (e.g., ISO 9000); furthermore, if widely adopted, they could provide a rational basis for continuing confidence in the results of a SCAMPI appraisal over time without incurring the cost of repeated SCAMPI appraisals. Thus while there is no requirement for periodic reviews, they do represent a best practice.

Myth 13
Formal structured Functional Area Representatives (FAR) interviews carry more weight in a SCAMPI than informal opportunistic interviews
.

It is less disruptive and far less expensive to conduct interviews as needed. This mode is encouraged by the SCAMPI method.

Myth 14
Every SCAMPI lead appraiser has “tenure” in the SEI system.

First, SCAMPI lead appraisers must maintain their skills to be re-licensed. This currently requires them to participate on at least two SCAMPI Class A appraisals and lead at least one during the authorization period. Licenses do expire, and certain actions may result in a license being withdrawn.

About the Author

Mike Phillips is the Director of Special Projects at the SEI, a position created to lead the Capability Maturity ModelIntegration (CMMI_ project for the SEI. He was previously responsible for transition-enabling activities at the SEI.

Prior to his retirement as a colonel from the Air Force, he managed the $36B development program for the B-2 in the B-2 SPO and commanded the 4950th Test Wing at Wright-Patterson AFB, OH. In addition to his bachelor’s degree in astronautical engineering from the Air Force Academy, Phillips has masters degrees in nuclear engineering from Georgia Tech, in systems management from the University of Southern California, and in international affairs from Salve Regina College and the Naval War College.

Please note that current and future CMMI research, training, and information has been transitioned to the CMMI Institute, a wholly-owned subsidiary of Carnegie Mellon University.

Find Us Here

Find us on Youtube  Find us on LinkedIn  Find us on twitter  Find us on Facebook

Share This Page

Share on Facebook  Send to your Twitter page  Save to del.ico.us  Save to LinkedIn  Digg this  Stumble this page.  Add to Technorati favorites  Save this page on your Google Home Page 

For more information

Contact Us

info@sei.cmu.edu

412-268-5800

Help us improve

Visitor feedback helps us continually improve our site.

Please tell us what you
think with this short
(< 5 minute) survey.