Choosing a Supplier: Due Diligence and CMMI Levels1

NEWS AT SEI

Author

Mike Phillips

This library item is related to the following area(s) of work:

Process Improvement
CMMI

This article was originally published in News at SEI on: February 1, 2004

We often find that customers who are considering the acquisition of software intensive products and services demand a CMM/CMMI level from potential suppliers as a perceived guarantee of a provider’s capabilities. We do believe that higher maturity organizations are more likely to deliver higher quality software intensive systems faster than those with less process discipline. We also believe that acquiring organizations should behave like those doing corporate acquisitions: they should apply “due diligence” to the choice of supplier.

For those who may be unfamiliar with such efforts, due diligence often includes a report containing information such as a Dun & Bradstreet financial review, debt analysis, and even turnover rate, as well as relevant past performance references. (A prospective developer with a track record of displeased clients must be identified prior to any contract awards regardless of an advertised CMMI benchmark claim.) Other acquiring organizations may not call their investigations “due diligence,” but instead characterize them as “risk appraisals” for their proposed buy. For the purposes of this column, we’ll consider these approaches to be the same.

The reasons for suggesting due diligence go well beyond stories about CMM or CMMI levels; but a conversation with John Vu of Boeing, as I began preparing this column, was too instructive to pass up. John mentioned that in his travels through Asia, he came upon a bag of rice in a store. On the bag, quality was assured by both the “Underwriters Laboratory” symbol ... and “CMM Level 3”!!!

Will Hayes, SEI’s lead for appraisal quality assurance, has developed a number of questions that can be used to gain more confidence in the choice of a supplier. These are listed below. Not all will be appropriate in every case, but they may help with the due-diligence process.

Focus first on the company….

Was a SCAMPI Class A appraisal performed on the organization?

We sometimes hear of CMMI levels being claimed using various methods. Some of these are homegrown, while others have been well developed both within the United States and internationally. The SEI cannot assure the quality of appraisals done outside the scope of its training and quality assurance. Thus the only CMMI levels that the SEI can attest to are those performed using the SCAMPI Class A method by an appraisal team led by an SEI-authorized Lead Appraiser. The appraisal team of at least four individuals may be from within the organization, from outside of the organization, or a mix of both.

Other appraisal methods may be useful for determining process strengths and weaknesses within the organization and may assist in the due-diligence effort–but a level rating from anything other than a SCAMPI will not be within SEI purview.

Where are the appraisal reports?

This question provides assurance that an appraisal was actually performed. The SEI receives the appraisal disclosure statement, which is a top-level look at elements of the SCAMPI. Further, both CMM and CMMI appraisals deliver to the organization the results of the investigation, with strengths and opportunities for improvement. The breadth of organizational coverage (covered in the appraisal plan), staff interviewed, and findings (addressed in the final report) may all be valuable to a discerning acquirer. These cannot be obtained from the SEI because of our commitment to confidentiality, but these documents may well be available from the appraised organization. (The prospective customer needs to recognize the responsibility to protect proprietary information. This can be assured by executing a non-disclosure agreement.)

The SEI has committed to make elements of the SCAMPI appraisal disclosure statement available to the public if the sponsor of the appraisal requests this in writing. This approach is new, so an organization’s absence from this list should not be viewed negatively. Some organizations may consider their results sensitive internal information and therefore choose not to allow public release. The Web pages only summarize appraisal results, but may give some of the answers to the questions below.

What part of the company was appraised?

Often appraisals focus on a portion of the overall enterprise. This is called the “organizational scope” of the appraisal. For various reasons, the portion of the enterprise of interest to the acquisition organization may or may not be represented by the results of the appraisal. In an ideal world, the development teams of interest will have been on actual projects that were specifically part of the appraisal. For example, a multi-site appraisal may have been performed, but the projects of interest to the acquisition organization may have been at a site not visited by the appraisal team. In addition, most appraisals use project sampling. Thus projects of greatest interest to the acquirer might not have been interviewed, but were still considered “covered,” if the appraisal team has confidence that the processes checked are “institutionalized” across that part of the organization.

Were the types of projects in the appraisal relevant to your business?

The CMMI material is indifferent to the types of development being accomplished. There can be greater confidence in the due-diligence investigation when the projects included in the appraisal are closely aligned with your needs.

Are any of the “not applicable” process areas important to your decision?

In addition to organizational scope, the appraisal allows choices in model scope. Differences in choice of coverage may be important to the investigation. For example, effective partnerships with other organizations may be essential for a large, complex development. The Integrated Supplier Management (ISM) process area (and the CMMI-SE/SW/IPPD/SS model) addresses this need. Did the organization choose to appraise its practices in this area? (Supplier Agreement Management, at Level 2, provides the initial capability, but some organizations have indicated that this area is “not applicable” as well.)

Did the organization choose to appraise only software engineering or systems engineering?

If capabilities associated with integrated operations, such as those characterized under the Integrated Process and Product Development (IPPD) category, are needed in a multi-organizational effort, has the organization measured itself against the IPPD elements? Sometimes what has been excluded from coverage is as important as what has been included.

When was the appraisal?

The SEI does not require appraisals on any timeline. The U.S. Department of Defense, in earlier policy statements, described a two year “acceptability” of appraisal results. (The two year time frame is also addressed if the CMMI appraisal is being registered, a function the SEI has offered the DoD when trained government participants are on the appraisal.) The length of time since the last appraisal must be considered within the context of the nature of the business environment. In a stable domain, older results might still be relevant; but in a dynamic environment, often with organizational changes and mergers and acquisitions, confidence in continued relevance diminishes. Further, since the intent of the levels is continuous process improvement, an aging appraisal may suggest that progress may be at risk.

How long did it take the organization to move up through the levels?

This question should remind the investigator that improvement does take time and that rapid movement to high maturity levels is neither easily accomplished nor frequently observed. (See the last question on high-maturity ratings below.)

Have there been significant organizational changes since the appraisal?

Mergers deserve special mention here. Acquiring a high-maturity company does not automatically confer that level on the newly merged organization. In fact, when two high-maturity organizations merge, some time is often required to integrate process approaches across the new enterprise.

Changes in senior leadership–even without merger and acquisition activity–can cause significant change in the commitment to process discipline. Staff continuity since the appraisal could therefore be worthwhile to check.

How does the organization train its people for process excellence?

This question follows those above. Effective training programs sustain the commitment to process excellence and are essential for effective organizational growth. Without this commitment, sustainment of existing levels as well as further improvement come into question.

Additional information for the due diligence can be obtained by searching the Web for publications of the developer’s strategic plans. Typically, the information in these plans merges the process improvement initiatives and required training for as long as five years.

…then on the Lead Appraiser...

Who was the Lead Appraiser?

The SEI maintains a list of all of the Lead Appraisers authorized to perform a benchmark “Class A” appraisal. Checking the name on the appraisal report with the list is a simple way to build confidence.

Was the Lead Appraiser independent or from within the organization appraised?

The SEI does not require that appraisals be conducted by Lead Appraisers from outside the organization. Many organizations have determined that to avoid concerns about objectivity, they hire independent appraisers for the benchmarking appraisal. Others prefer an appraiser from within the company, but, for example, from a function charged with assuring organizational performance of operational elements of the company.

Did the Lead Appraiser guide the improvement effort?

While we understand that Lead Appraisers are experts at process improvement, extensive involvement in preparing the organization for the appraisal, and then leading the appraisal, puts the appraiser in a difficult position. From a due-diligence perspective, the investigators might wish to dig deeper if this phenomenon is evident.

…and then for “high-maturity” (Level 4 & 5) claims…

If the organization was given a high maturity rating, what experience did the Lead Appraiser have in high-maturity appraisals?

All Lead Appraisers receive standard training at the SEI (Introduction to CMMI, Intermediate CMMI, Lead Appraiser Training) and all Lead Appraisers are observed (by an SEI representative) before they are allowed to lead an appraisal on their own. However, there is a difference in appraisals of low-maturity organizations (1,2,3) and high-maturity organizations (4,5). Organizations behave differently at high maturity, and it takes an experienced appraiser to recognize the cultural differences. It is important to determine if the Lead Appraiser has had previous experience on high-maturity appraisal teams or has led a previous high-maturity appraisal (an appraisal where a level 4 or level 5 was recognized).

1 Some of these questions were published in the March 1, 2004, CIO Magazine, in the sidebar to "Bursting the CMM Hype" by Christopher Koch, after a discussion with Will Hayes, quality manager for the SEI Appraisal Program. For more information, go to http://www.cio.com/archive/030104/cmm_sidebar_2.html

About the Author

Mike Phillips is the Director of Special Projects at the SEI, a position created to lead the Capability Maturity Model Integration (CMMI) project for the SEI. He was previously responsible for transition-enabling activities at the SEI.

Prior to his retirement as a colonel from the Air Force, he managed the $36B development program for the B-2 in the B-2 SPO and commanded the 4950th Test Wing at Wright-Patterson AFB, OH. In addition to his bachelor’s degree in astronautical engineering from the Air Force Academy, Phillips has masters degrees in nuclear engineering from Georgia Tech, in systems management from the University of Southern California, and in international affairs from Salve Regina College and the Naval War College.

Please note that current and future CMMI research, training, and information has been transitioned to the CMMI Institute, a wholly-owned subsidiary of Carnegie Mellon University.

Find Us Here

Find us on Youtube  Find us on LinkedIn  Find us on twitter  Find us on Facebook

Share This Page

Share on Facebook  Send to your Twitter page  Save to del.ico.us  Save to LinkedIn  Digg this  Stumble this page.  Add to Technorati favorites  Save this page on your Google Home Page 

For more information

Contact Us

info@sei.cmu.edu

412-268-5800

Help us improve

Visitor feedback helps us continually improve our site.

Please tell us what you
think with this short
(< 5 minute) survey.