Conducting a SCAMPI Appraisal
This topic contains questions and answers about conducting a SCAMPI appraisal in your organization.
Contents:
- Is it considered a conflict of interest if an external consultant is hired by a company for the implementation of a process improvement program based on CMMI and a person from the same consulting company performs a SCAMPI appraisal on the company? Is there something that could be done to make this situation acceptable to the SEI or is it a situation that should be avoided completely?
- How do I select an SEI Partner?
- What is the typical size of a SCAMPI A appraisal team?
- Must all projects apply the same standard process during a SCAMPI appraisal?
- Must the projects appraised in a SCAMPI A appraisal cover the entire product lifecycle?
- Can interview questions asked during a SCAMPI be interpreted?
- Can a SCAMPI appraisal cover just software engineering or just systems engineering?
- Is there a process to register appraisals?
- How do I report the improper application of a SCAMPI appraisal?
- How are informative model elements considered during an appraisal?
- What does the Lead Appraiser report to the SEI after a SCAMPI appraisal?
- Why should I report my appraisal results to the SEI?
- Are appraisal data kept and verified by the SEI?
- When does the SEI audit a SCAMPI appraisal?
- Are interviews considered adequate evidence in some circumstances?
- What are the guidelines for evidence in SCAMPI appraisals of multi-site organizations?
- Can I add to the SCAMPI method during an appraisal?
- Can multiple Appraisers receive credit for leading the same SCAMPI appraisal?
- Can any SCAMPI Lead Appraiser or SCAMPI B&C Team Leader be an observer?
- Are observers allowed in a SCAMPI appraisal?
- Does the appraisal sponsor have to be notified that a SCAMPI appraisal is observed?
Is it considered a conflict of interest if an external consultant is hired by a company for the implementation of a process improvement program based on CMMI and a person from the same consulting company performs a SCAMPI appraisal on the company? Is there something that could be done to make this situation acceptable to the SEI or is it a situation that should be avoided completely?
Any time an employee of an organization is in the position of
appraising work done by other members of their own organization, there
is a potential for conflict of interest. The lead appraiser should do
his or her best to avoid these situations.
Refer to the Code of Professional Conduct on page 8 where it states, "Ensure that the inherent conflicts of interest that occur when your Authorized or Certified Professionals and Candidates are placed in the role of evaluating their own work or your organizations work are disclosed to all affected parties (see Section 4.2) and take action to minimize the impact or eliminate the conflict."
On page 13 in Section 4.2, Conflict
of Interest Disclosure,
it also states, "When a situation involving a conflict of interest is
inherent or cannot be avoided, disclose the conflict to the affected
parties, including your own organization, the Client, and the SEI. The
disclosure report is available at http://www.sei.cmu.edu/partners/conflict-form.html."
If this situation is unavoidable, there are ways to mitigate
the risk:
- When feasible, the lead appraiser delivering the appraisal should be from a different part of the consulting company (e.g., different division) than the person who delivered the consulting, and should also report to a different manager than the person who delivered the consulting.
- Utilize appraisal team members who are independent from the organization being appraised, perhaps external to the company being appraised and from the consulting company.
- The conflict should also be addressed in the risks and mitigation section of the appraisal plan so that a quality review will see that it was disclosed and addressed.
How do I select an SEI Partner?
Visit the SEI Partner Network Directory at http://partner-directory.sei.cmu.edu/ to search for SEI Partner organizations. You can search for Partners by name, services provided, or by location. Many Partners provide CMMI training service, SCAMPI Lead Appraiser services, and additional process improvement assistance.
The SEI does not recommend a particular SEI Partner organization or their employees. Your decision should be based on discussions with the partners that you contact. If you have a question about whether an organization you are working with is an authorized member of the SEI Partner Network, check the directory. If they do not appear, contact SEI Customer Relations at customer-relations@sei.cmu.edu for more information.
return to topWhat is the typical size of a SCAMPI A appraisal team?
The SCAMPI method requires a minimum team size of four (the SCAMPI Lead Appraiser counts toward the team size); the recommended upper limit on team size is nine.
Based on a sample of recently completed SCAMPI A appraisals, by maturity level rendered (from maturity level 1 through level 5), the average team size was seven, six, seven, nine and eight respectively. However, the differences in these numbers do not appear to be statistically significant or recommend an ideal size for the appraisal team.
return to topMust all projects apply the same standard process during a SCAMPI appraisal?
Depending on where an existing project is in its lifecycle, it may be a while before the project can begin to utilize the new standard processes, or it may even be impossible for the project to use the new standard process. For example, for a project in which the design was already complete at the time of the introduction of the new standard processes, there will be no experience (or artifacts) reflecting use of these new standard processes. Similarly, it is not expected that once a new process is established that a project will redo prior work using the new process; however, new work done after the establishment of the new process would normally be expected to follow the new process.
The sample of projects selected for examination during an appraisal should be representative of that organization. However, this may mean selecting one or more projects which have not made full use of the new standard processes. Rather than investigate these, and report on both new and older projects in the appraisal results, it may make more sense to define the scope of the organization that is the focus of the appraisal (the "organizational unit") and to exclude older projects, particularly if they represent an older line of business. These decisions must be documented in the Appraisal Input and Appraisal Disclosure Statement.
In any case, it would be reasonable to expect over time that most projects will eventually use the new standard processes. The appraisal team should account for risks it sees as inherent in the appraisal situation.
return to topMust the projects appraised in a SCAMPI A appraisal cover the entire product lifecycle?
The lifecycle covered in a SCAMPI A appraisal is defined by the scope of the CMMI model process areas selected for the appraisal.
The lifecycle selected does not need to be covered by each representative project; it can be covered collectively by the identified representative projects.
SCAMPI is based on the fundamental premise that to make valid inferences about the extent to which best practices are implemented at the organizational level, a representative number of instantiations of the practice must be examined. For practices that relate to a project activity (as opposed to an organizational activity such as engineering process groups), a representative sample of projects is identified as part of the appraisal planning and preparation phase.
In appraisals where the reference model scope includes any project-related process area, the organizational scope must include at least one focus project. If the organizational unit includes more than three projects, then the organizational scope must include sufficient focus projects and non-focus projects to generate at least three instances of each practice in each project-related PA in the model scope of the appraisal.
return to topCan interview questions asked during a SCAMPI be interpreted?
There are no method requirements that restrict or constrain the wording of interview questions. The appraisal team seeks to deepen their understanding of, or confirm, existing objective evidence relating to the implementation of a model practice. Accordingly, means that will contribute to these objectives are allowed and encouraged, including clarifications offered by members of the appraisal team, as long as these do not otherwise conflict with previously agreed-on team ground rules such as not "leading" the interviewee to a particular response.
It is important to keep in mind that the SCAMPI A appraisal method is a benchmarking appraisal method. It is applied most effectively and efficiently in contexts where there is at least some awareness and understanding of model concepts and terminology. However, the onus is on the appraisal team to understand how the CMMI model relates to the organization and to bridge the gap between model language and the organization's language.
return to topCan a SCAMPI appraisal cover just software engineering or just systems engineering?
Yes. CMMI models permit the flexible application of Engineering practices (i.e., those contained in the Requirements Management, Requirements Development, Technical Solution, Product Integration, Verification, and Validation process areas) to software engineering, systems engineering, or other product engineering functions. Each organization selects a CMMI model and the parts of the organization to participate in the process improvement program (i.e., the scope).
The SCAMPI appraisal method can be used to perform discipline-specific appraisals by simply selecting the scope of the appraisal using the "organizational unit" concept and using the appropriate CMMI model. An organizational unit typically has an identifiable senior manager, deploys one or more processes that have a coherent process context, and operates within a coherent set of business objectives. An organizational unit is typically part of a larger organization, although in a small organization, the organizational unit may be the whole organization. Such an appraisal is not termed "CMMI-SE" or "CMMI-SW" as in the past, nor can "SE" or "SW" be appended to the new formal model definition (i.e., CMMI-DEV/SE is not allowed).
As part of every appraisal, an Appraisal Disclosure Statement (ADS) must be completed. This statement must indicate not only the CMMI model used for the appraisal, but also the organizational unit(s) included in the appraisal. This statement clearly communicates the nature of the appraisal, including whether software engineering or systems engineering processes of the organization were appraised.
return to topIs there a process to register appraisals?
At this time, there is not a formalized "registered appraisal" process that is endorsed by the senior leadership at the Office of the Secretary of Defense level or the senior management in the acquisition communities of the Services. Further, there has been no uniform effort to train qualified people throughout the Services or Defense Contract Management Agency (DCMA) to support a "registered appraisal" process. However, some users of SCAMPI appraisals have found the concept to have merit and has been piloted in several specific environments.
Therefore, a registration service for cooperative government/industry appraisals is now being offered by the SEI for the purpose of encouraging reuse of appraisals as part of applicable U.S. government source selections. Registered appraisals may be submitted by an organization to the U.S. government (e.g., the DoD) in lieu of independent evaluations. These registrations are valid for no more than two years.
As this is not formalized, any participation would be completely voluntary for both government and industry. For example, a government organization may or may not choose to accept a particular registered appraisal, or an industry organization may or may not choose to submit a registered appraisal for consideration on a particular source selection.
Registration of an appraisal means that the appraisal report package submitted to the SEI contains the artifacts required by the SCAMPI A method, meets other criteria considered essential to classifying the appraisal as a SCAMPI A appraisal, that an SEI-authorized lead appraiser led the appraisal, and that at least one of the two government participants was an SEI-authorized lead appraiser or candidate lead appraiser. Both government participants must have been fully trained in the method and model used in the appraisal. The government personnel may not be "observers," as these are not allowed in a formal SCAMPI appraisal.
The registration does not certify or otherwise validate the maturity level or other findings reported in the appraisal report package nor does it guarantee the past or future performance of the organizational entity.
For further information about this registration service offered at the SEI, contact SEI Customer Relations at 412-268-5800 or email customer-relations@sei.cmu.edu
return to topHow do I report the improper application of a SCAMPI appraisal?
The following feedback mechanisms currently exist:
For SCAMPI A appraisals, the appraisal sponsor, appraisal team leader, and appraisal team members are required to provide feedback to the CMMI Steward (the SEI). More broadly, the SEI has established an email alias through which quality-related issues for the SCAMPI method can be reported: scampi-quality@sei.cmu.edu.
If the non-model-related issue affects one of the CMMI products, then a change request can be submitted. You can submit change requests and comments about CMMI any time you like using the CMMI User Feedback Process. Use one of the three change request forms or the comment form to submit feedback based on your use of the CMMI Product Suite. You can access the forms at http://www.sei.cmu.edu/cmmi/models/change-requests.html.
return to topHow are informative model elements considered during an appraisal?
SCAMPI Lead Appraisers (and their teams) are taught that the full model (from cover to cover) should be considered the basis for understanding the model, and the process area goals are the only normative rating component. Practice statements are designed to encapsulate or summarize the intent of a practice. To fully comprehend the intent of a practice, the context established by the full CMMI model must be understood; this includes in particular subpractices, the front matter, and the glossary.
When implementing CMMI level 4-5 practices in an organization and when appraising an organization against the CMMI practices at levels 4-5, special attention must be given to the informative material at these levels in a CMMI model. For more information about CMMI Level 4-5 Interpretation and Implementations, see http://www.sei.cmu.edu/cmmi/adoption/cmmilevels45.html.
return to topWhat does the Lead Appraiser report to the SEI after a SCAMPI appraisal?
A SCAMPI A appraisal must be set up on the SEI's SCAMPI Appraisal System (SAS) prior to delivery. The information to be reported includes the name of the SEI Partner, name of the Lead Appraiser, name of the organization to be appraised, location, projected onsite date, and names of the team members.
Once the appraisal is complete, the following artifacts of the appraisal must be reported to the SEI in English:
- Appraisal input
- Appraisal plan
- Final findings briefing
- Appraisal Disclosure Statement (ADS)
- Process Appraisal Information System (PAIS) Record of entry form
- Appraisal feedback form
In addition, feedback forms are required from the appraisal team members and the appraisal sponsor.
SCAMPI B and C appraisals are less rigorous and require less setup and reporting. The following artifacts must be reported to the SEI steward in English for both a SCAMPI B and a SCAMPI C:
- Appraisal input
- Appraisal plan
- Final findings briefing
- Appraisal Disclosure Statement (ADS)
- Process Appraisal Information System (PAIS) Record of entry form
These reporting requirements are subject to change. For more information, see http://www.sei.cmu.edu/appraisal-program/reporting.html.
return to topWhy should I report my appraisal results to the SEI?
Reporting your organization's appraisal results helps to create a more complete picture of the state of CMMI-based process improvement methods. Understanding the current state of adoption of these methods allows the greater software and systems development community to accomplish the following:
- Examine improvement trends. One example is the Maturity Profile and its interactive version. Neither would be possible without the participation of appraised organizations.
- Improve existing or develop new models. Appraisal data helps to uncover trends in the adoption of an improvement method, often revealing global strengths and weaknesses.
- Support research publications.
- Allow Appraiser training, authorization, and renewal.
Further, if the results of the appraisal are to be announced publicly, used in a formal proposal, or announced outside of the appraised organization, but within the same company, then the appraisal results must be placed on the SEI's Public Appraisal Results Site (PARS).
return to topAre appraisal data kept and verified by the SEI?
SCAMPI Lead Appraisers and SCAMPI B&C Team Leaders are required to submit a data package to the SEI following the completion of SCAMPI appraisals they lead. The SEI uses these data for research and reporting purposes; they are not made public in any way that would compromise the confidentiality of the source.
All data reported to the SEI from appraisals are treated as confidential to the organization submitting the data. This means that access to these data is tightly controlled. Only SEI personnel with a need to know are granted access to identified data.
return to topWhen does the SEI audit a SCAMPI appraisal?
The SEI audits all appraisal artifacts provided by SCAMPI Lead Appraisers and SCAMPI B&C Team Leaders. An on-site audit of an appraisal may be initiated by one or more of the following:
- request of the appraisal sponsor
- credible reports of misconduct
- failure to fulfill SCAMPI Lead Appraiser or SCAMPI B&C Team Leader responsibilities
- random selection
Additionally, the SEI will soon verify the results of all appraisals prior to allowing the appraised organization to release the results outside of the appraised organization. Such verification will be accomplished within 30 days of submittal of the results.
return to topAre interviews considered adequate evidence in some circumstances?
No. Artifact evidence is a requirement of the SCAMPI A method. There must be a direct artifact for each specific practice appraised. A direct artifact is one that is a direct result of the performance of the process, for example a work breakdown structure (WBS). Those direct artifacts then must be supported by either an affirmation or an indirect artifact. An indirect artifact is one that is created as a result of the process being performed (e.g., the attendance sign-in sheet for meetings where the WBS was created). Affirmations are given in interviews during the appraisal.
The typical work products that are identified within the description of each specific practice in the CMMI models suggest possible artifacts that might result from and demonstrate its implementation. For small-team scenarios, the level of detail and formality of these artifacts should be appropriate for the need but still give the appraisal team confidence that the relevant goals have been satisfied.
return to topWhat are the guidelines for evidence in SCAMPI appraisals of multi-site organizations?
There is no SCAMPI A requirement for the appraisal team to have a physical presence at each "satellite site;" however, having no visibility into a site is not recommended. At a minimum, some "face time" with remote employees is desirable even if it is at a distance via teleconferencing or videoconferencing.
return to topCan I add to the SCAMPI method during an appraisal?
The SCAMPI method is designed to be used in a wide variety of appraisal scenarios, so tailoring of the method has received a lot of attention. There is a great deal of flexibility provided to add things as long as they are consistent with the method itself and do not violate any current SCAMPI requirements.
return to topCan multiple Appraisers receive credit for leading the same SCAMPI appraisal?
No. Only the individual assigned the role of appraisal team leader can get credit even if there is some sharing of responsibilities with another SCAMPI Lead Appraiser or SCAMPI B&C Team Leader who happens to be a team member.
return to topCan any SCAMPI Lead Appraiser or SCAMPI B&C Team Leader be an observer?
No. Only SEI authorized observers can perform this function for SCAMPI A observations. SCAMPI B&C Team Leaders who have fulfilled the requirements to become observers may apply by sending their request to btl-observer@sei.cmu.edu where it will be evaluated.
return to topAre observers allowed in a SCAMPI appraisal?
No. Except for SEI-authorized observers, observers are not admitted as part of a SCAMPI A appraisal. According to the SCAMPI Method Definition Document, clause 1.3.3 (Roles and Responsibilities), "Due to the confidentiality required during an appraisal and the cohesiveness needed to participate in appraisal activities, observers are not permitted to participate in the appraisal processes."
return to topDoes the appraisal sponsor have to be notified that a SCAMPI appraisal is observed?
Yes. It is the responsibility of the candidate SCAMPI Lead Appraiser and candidate SCAMPI B&C Team Leader to inform the appraisal sponsor verbally and to record the facts about the observation in the appraisal input.
return to topIf you have a question that is not answered within this FAQ, check the CMMI FAQ Main Page or send email to an SEI person at cmmi-comments@sei.cmu.edu. You will receive a response within 48 hours.