SCAMPI Appraisal Rules
Answers questions about the details of a SCAMPI that involve what can and cannot be done during an appraisal
Contents:
Can a SCAMPI appraisal cover just software engineering or just systems engineering?
Can interview questions asked during a SCAMPI be interpreted?
Are interviews considered adequate evidence in some circumstances?
What are the guidelines for evidence in SCAMPI appraisals of multi-site organizations?
Can I add to the SCAMPI method during an appraisal?
Must all projects apply the same standard process during a SCAMPI appraisal?
Must the projects appraised in a SCAMPI A appraisal cover the entire product lifecycle?
What does the Lead Appraiser report to the SEI after a SCAMPI appraisal?
Can multiple Appraisers receive credit for leading the same SCAMPI appraisal?
Can any SCAMPI Lead Appraiser or SCAMPI B&C Team Leader be an observer?
Are observers allowed in a SCAMPI appraisal?
Does the appraisal sponsor have to be notified that a SCAMPI appraisal is observed?
Can a SCAMPI appraisal cover just software engineering or just systems engineering?
Yes. CMMI models permit the flexible application of Engineering practices (i.e., those contained in the Requirements Management, Requirements Development, Technical Solution, Product Integration, Verification, and Validation process areas) to software engineering, systems engineering, or other product engineering functions. Each organization selects a CMMI model and the parts of the organization to participate in the process improvement program (i.e., the scope).
The SCAMPI appraisal method can be used to perform discipline-specific appraisals by simply selecting the scope of the appraisal using the "organizational unit" concept and using the appropriate CMMI model. An organizational unit typically has an identifiable senior manager, deploys one or more processes that have a coherent process context, and operates within a coherent set of business objectives. An organizational unit is typically part of a larger organization, although in a small organization, the organizational unit may be the whole organization.
As part of every appraisal, an Appraisal Disclosure Statement (ADS) must be completed. This statement must indicate not only the CMMI model used for the appraisal, but also the organizational unit(s) included in the appraisal. This statement clearly communicates the nature of the appraisal, including whether software engineering or systems engineering processes of the organization were appraised.
return to top
Can interview questions asked during a SCAMPI be interpreted?
There are no method requirements that restrict or constrain the wording of interview questions. The appraisal team seeks to deepen their understanding of, or confirm, existing objective evidence relating to the implementation of a model practice. Accordingly, means that will contribute to these objectives are allowed and encouraged, including clarifications offered by members of the appraisal team, as long as these do not otherwise conflict with previously agreed-on team ground rules such as not "leading" the interviewee to a particular response.
It is important to keep in mind that the SCAMPI A appraisal method is a benchmarking appraisal method. It is applied most effectively and efficiently in contexts where there is at least some awareness and understanding of model concepts and terminology. However, the onus is on the appraisal team to understand how the CMMI model relates to the organization and to bridge the gap between model language and the organization's language.
return to topAre interviews considered adequate evidence in some circumstances?
No. Artifact evidence is a requirement of the SCAMPI A method. The typical work products that are identified within the description of each specific practice in the CMMI models suggest possible artifacts that might result from and demonstrate its implementation. For small-team scenarios, the level of detail and formality of these artifacts should be appropriate for the need but still give the appraisal team confidence that the relevant goals have been satisfied.
return to topWhat are the guidelines for evidence in SCAMPI appraisals of multi-site organizations?
There is no SCAMPI A requirement for the appraisal team to have a physical presence at each "satellite site;" however, having no visibility into a site is not recommended. At a minimum, some "face time" with remote employees is desirable even if it is at a distance via teleconferencing or videoconferencing.
return to topCan I add to the SCAMPI method during an appraisal?
The SCAMPI method is designed to be used in a wide variety of appraisal scenarios, so tailoring of the method has received a lot of attention. There is a great deal of flexibility provided to add things as long as they are consistent with the method itself. The SCAMPI Implementation Model, or SIM, is specifically designed to provide a way of thinking about SCAMPI (as well as other CMMI appraisal methods) in such a way as to facilitate tailoring and adaptation.
return to topMust all projects apply the same standard process during a SCAMPI appraisal?
Depending on where an existing project is in its lifecycle, it may be a while before the project can begin to utilize the new standard processes, or it may even be impossible for the project to use the new standard process. For example, for a project in which the design was already complete at the time of the introduction of the new standard processes, there will be no experience (or artifacts) reflecting use of these new standard processes.
The sample of projects selected for examination during an appraisal should be representative of that organization. However, this may mean selecting one or more projects which have not made full use of the new standard processes. Rather than investigate these, and report on both new and older projects in the appraisal results, it may make more sense to define the scope of the organization that is the focus of the appraisal (the "organizational unit") and to exclude older projects, particularly if they represent an older line of business. These decisions must be documented in the Appraisal Input and Appraisal Disclosure Statement.
In any case, it would be reasonable to expect over time that most projects will eventually use the new standard processes. The appraisal team should account for risks it sees as inherent in the appraisal situation.
return to topMust the projects appraised in a SCAMPI A appraisal cover the entire product lifecycle?
The lifecycle covered in a SCAMPI A appraisal is defined by the scope of the CMMI model process areas selected for the appraisal.
The lifecycle selected does not need to be covered by each representative project; it can be covered collectively by the identified representative projects.
SCAMPI is based on the fundamental premise that to make valid inferences about the extent to which best practices are implemented at the organizational level, a representative number of instantiations of the practice must be examined. For practices that relate to a project activity (as opposed to an organizational activity such as engineering process groups), a representative sample of projects is identified as part of the appraisal planning and preparation phase.
In practice, it may not be possible to find a set of projects that have progressed to the point where they each (separately) have traversed the selected lifecycle; consequently, SCAMPI Lead Appraisers, working with the appraisal sponsor and designated representatives, identify a set of projects that, when taken as a whole, cover the lifecycle. The key consideration to making this an effective and valid strategy is whether the projects considered for each practice are truly representative of the remainder of the organizational unit being appraised.
return to topWhat does the Lead Appraiser report to the SEI after a SCAMPI appraisal?
A SCAMPI A appraisal must be set up on the SEI's SCAMPI Appraisal System (SAS) prior to delivery. The information to be reported includes the name of the SEI Partner, name of the Lead Appraiser, name of the organization to be appraised, location, projected onsite date, and names of the team members.
Once the appraisal is complete, the following artifacts of the appraisal must be reported to the SEI in English:
- Appraisal input
- Appraisal plan
- Final findings briefing
- Appraisal Disclosure Statement (ADS)
- Process Appraisal Information System (PAIS) Record of entry form
- Appraisal feedback form
In addition, feedback forms are required from the appraisal team members and the appraisal sponsor.
SCAMPI B and C appraisals are less rigorous and require less setup and reporting. The following artifacts must be reported to the SEI steward in English for both a SCAMPI B and a SCAMPI C:
- Appraisal input
- Appraisal plan
- Final findings briefing
- Appraisal Disclosure Statement (ADS)
- Process Appraisal Information System (PAIS) Record of entry form
These reporting requirements are subject to change. For more information, see http://www.sei.cmu.edu/appraisal-program/reporting.html.
return to topCan multiple Appraisers receive credit for leading the same SCAMPI appraisal?
No. Only the individual assigned the role of appraisal team leader can get credit even if there is some sharing of responsibilities with another SCAMPI Lead Appraiser or SCAMPI B&C Team Leader who happens to be a team member.
return to topCan any SCAMPI Lead Appraiser or SCAMPI B&C Team Leader be an observer?
No. Only SEI authorized observers can perform this function for SCAMPI A observations. SCAMPI B&C Team Leaders who have fulfilled the requirements to become observers may apply by sending their request to btl-observer@sei.cmu.edu where it will be evaluated.
return to topAre observers allowed in a SCAMPI appraisal?
No. Except for SEI-authorized observers, observers are not admitted as part of a SCAMPI A appraisal. According to the SCAMPI Method Definition Document, clause 1.3.3 (Roles and Responsibilities), "Due to the confidentiality required during an appraisal and the cohesiveness needed to participate in appraisal activities, observers are not permitted to participate in the appraisal processes."
return to topDoes the appraisal sponsor have to be notified that a SCAMPI appraisal is observed?
Yes. It is the responsibility of the candidate SCAMPI Lead Appraiser and candidate SCAMPI B&C Team Leader to inform the appraisal sponsor verbally and to record the facts about the observation in the appraisal input.
return to topIf you have a question that is not answered within this FAQ, check the CMMI FAQ Main Page or send email to an SEI person at cmmi-comments@sei.cmu.edu. You will receive a response within 48 hours.