Reporting Appraisal Results
This topic contains questions and answers about what to expect as the result of an appraisal.
Contents:
- What does the Lead Appraiser report to the SEI after a SCAMPI appraisal?
- Can multiple Appraisers receive credit for leading the same SCAMPI appraisal?
- Does an appraisal always result in a maturity level?
- Does my organization receive a certificate for achieving a maturity level rating?
- Does the government use maturity levels in making acquisition decisions?
- Is there a process to register appraisals?
- Do Lead Appraisers issue certificates to their students?
- How does my organization receive CMMI certification?
- How do I report the improper application of a SCAMPI appraisal?
- What is a maturity profile?
- Which organizations have achieved a CMMI maturity level?
- Does the SEI monitor or certify appraisals?
- Who should report appraisals?
- Why should I report my appraisal results to the SEI?
- Are appraisal data kept and verified by the SEI?
What does the Lead Appraiser report to the SEI after a SCAMPI appraisal?
A SCAMPI A appraisal must be set up on the SEI's SCAMPI Appraisal System (SAS) prior to delivery. The information to be reported includes the name of the SEI Partner, name of the Lead Appraiser, name of the organization to be appraised, location, projected onsite date, and names of the team members.
Once the appraisal is complete, the following artifacts of the appraisal must be reported to the SEI in English:
- Appraisal input
- Appraisal plan
- Final findings briefing
- Appraisal Disclosure Statement (ADS)
- Process Appraisal Information System (PAIS) Record of entry form
- Appraisal feedback form
In addition, feedback forms are required from the appraisal team members and the appraisal sponsor.
SCAMPI B and C appraisals are less rigorous and require less setup and reporting. The following artifacts must be reported to the SEI steward in English for both a SCAMPI B and a SCAMPI C:
- Appraisal input
- Appraisal plan
- Final findings briefing
- Appraisal Disclosure Statement (ADS)
- Process Appraisal Information System (PAIS) Record of entry form
These reporting requirements are subject to change. For more information, see http://www.sei.cmu.edu/appraisal-program/reporting.html.
return to top
Can multiple Appraisers receive credit for leading the same SCAMPI appraisal?
No. Only the individual assigned the role of appraisal team leader can get credit even if there is some sharing of responsibilities with another SCAMPI Lead Appraiser or SCAMPI B&C Team Leader who happens to be a team member.
return to topDoes an appraisal always result in a maturity level?
No. An appraisal does not necessarily result in a maturity level rating.
The SCAMPI A method can be enacted as either a staged or continuous process improvement model. When using a staged implementation, a maturity level can be awarded, but is not required. When using a continuous implementation, the model may produce capability levels for appraised process areas. A maturity level can be established through the continuous method by following the rules for Equivalent Staging. Your SCAMPI Lead Appraiser is trained in this activity.
return to topDoes my organization receive a certificate for achieving a maturity level rating?
The SEI does not validate the appraisal results of an organization, and therefore does not provide a certificate or any other artifact that acknowledges the appraisal results. However, this does not prevent the SEI Partner that performs an appraisal from presenting a certificate with the appraisal results to the organization.
The SEI does encourage standardized reporting of selected appraisal results so as to provide a reasonable basis for drawing inferences from reported appraisal results. For example, the SCAMPI method has a mandatory reporting requirement (the Appraisal Disclosure Statement). However, the use of the SEI logo on reporting forms is not permitted as it could be misconstrued to mean that the SEI verifies the accuracy of the information reported.
return to topDoes the government use maturity levels in making acquisition decisions?
Some government organizations consider maturity level ratings when making their acquisition decisions. Some of these organizations accept the results of appraisals conducted by the organization being evaluated for a contract. Other organizations request government-run evaluations.
return to topIs there a process to register appraisals?
At this time, there is not a formalized "registered appraisal" process that is endorsed by the senior leadership at the Office of the Secretary of Defense level or the senior management in the acquisition communities of the Services. Further, there has been no uniform effort to train qualified people throughout the Services or Defense Contract Management Agency (DCMA) to support a "registered appraisal" process. However, some users of SCAMPI appraisals have found the concept to have merit and has been piloted in several specific environments.
Therefore, a registration service for cooperative government/industry appraisals is now being offered by the SEI for the purpose of encouraging reuse of appraisals as part of applicable U.S. government source selections. Registered appraisals may be submitted by an organization to the U.S. government (e.g., the DoD) in lieu of independent evaluations. These registrations are valid for no more than two years.
As this is not formalized, any participation would be completely voluntary for both government and industry. For example, a government organization may or may not choose to accept a particular registered appraisal, or an industry organization may or may not choose to submit a registered appraisal for consideration on a particular source selection.
Registration of an appraisal means that the appraisal report package submitted to the SEI contains the artifacts required by the SCAMPI A method, meets other criteria considered essential to classifying the appraisal as a SCAMPI A appraisal, that an SEI-authorized lead appraiser led the appraisal, and that at least one of the two government participants was an SEI-authorized lead appraiser or candidate lead appraiser. Both government participants must have been fully trained in the method and model used in the appraisal. The government personnel may not be "observers," as these are not allowed in a formal SCAMPI appraisal.
The registration does not certify or otherwise validate the maturity level or other findings reported in the appraisal report package nor does it guarantee the past or future performance of the organizational entity.
For further information about this registration service offered at the SEI, contact SEI Customer Relations at 412-268-5800 or email customer-relations@sei.cmu.edu
return to topDo Lead Appraisers issue certificates to their students?
Yes. They are encouraged to do this so that students have a record of their training. However, the use of the SEI's logo on such certificates is not permitted.
return to topHow does my organization receive CMMI certification?
The SEI does not certify the results of any appraisal nor is there an official accreditation body for CMMI. True certification of appraisal results would involve the ongoing monitoring of organizations' capabilities, a shelf life for appraisal results, and other administrative elements.
The SEI does not have a defined requirement for periodic follow-up after appraisals, nor does it accept legal responsibility for the performance of appraised organizations. All of these characteristics are required for a program that would provide certification of appraisal results.
The SEI urges customers to request a copy of an Appraisal Disclosure Statement (ADS) when evaluating suppliers. The ADS not only documents the achieved maturity level or capability level profile but also documents which parts of the organization were appraised.
Organizations can report their CMMI-based achievements to the SEI to be published on the List of Published SCAMPI Appraisal Results Web page. See that page for more information.
For more information on appraisal methods and for a directory of authorized appraisers who can perform them, see the SEI Appraisal Program Web page at http://www.sei.cmu.edu/appraisal-program/index.html.
return to topHow do I report the improper application of a SCAMPI appraisal?
The following feedback mechanisms currently exist:
For SCAMPI A appraisals, the appraisal sponsor, appraisal team leader, and appraisal team members are required to provide feedback to the CMMI Steward (the SEI). More broadly, the SEI has established an email alias through which quality-related issues for the SCAMPI method can be reported: scampi-quality@sei.cmu.edu.
If the non-model-related issue affects one of the CMMI products, then a change request can be submitted. You can submit change requests and comments about CMMI any time you like using the CMMI User Feedback Process. Use one of the three change request forms or the comment form to submit feedback based on your use of the CMMI Product Suite. You can access the forms at http://www.sei.cmu.edu/cmmi/models/change-requests.html.
return to topWhat is a maturity profile?
A maturity profile is a slide presentation that shows how an improvement model (e.g., CMMI) is being adopted worldwide based on appraisal results reported by SEI-authorized appraisers. Data are presented in a series of graphs and bar charts by organization type, size, location, maturity level, and other variables.
The SEI has produced a maturity profile for the Software CMM since 1994. The first maturity profile for CMMI was released in 2003.
For more information or to download a copy of the most current CMMI maturity profile, see the maturity profiles page at http://www.sei.cmu.edu/appraisal-program/profile/.
return to topWhich organizations have achieved a CMMI maturity level?
The SEI does not publish the names of specific individuals, organizations, or projects without permission from the appraisal sponsor.
The SEI maintains a public database of Published SCAMPI Appraisal Results at http://sas.sei.cmu.edu/pars/. Once an appraisal has passed quality checkpoints, an authorization form is sent to the Lead Appraiser. When an organization wishes its appraisal results to be published on the SEI Web site, the appraisal sponsor must authorize it by signing and returning the authorization form. Once this form is received the appraisal detail will be displayed.
Appraisal information reported to the SEI is treated as proprietary and confidential. An appraisal's results are owned by the appraisal sponsor (usually a person within the organization that was appraised). The sponsor may publicize this information at his or her discretion.
If the organization prefers to formalize the confidentiality of the appraisal being reported to the SEI, the Lead Appraiser can use the Non-Disclosure Letter and the SEI Non-Disclosure Agreement.
return to topDoes the SEI monitor or certify appraisals?
No. Beyond the confidentiality issue, the only firsthand knowledge the SEI has of an appraisal--other than the occasional appraisal in which an SEI staff member participates--is that the appraisal has been performed by a person authorized by the SEI (i.e., a Lead Appraiser) to perform the appraisal.
This means that the Lead Appraiser has permission to use SEI materials and that their SEI authorization is in good standing. Although prerequisite qualifications, random audits, and other mechanisms are in place to monitor a Lead Appraiser's performance, the SEI does not have the resources or mission to closely monitor every Lead Appraiser's performance. The SEI's Appraisal Team performs quality reviews of appraisal records reported to assure completeness and correctness of data submitted. There is a method compliance test performed on appraisal reports submitted to assure all phases of the appraisal have been performed and appropriate information has been provided to make the appraisal's circumstances understandable.
return to topWho should report appraisals?
The SEI encourages every organization who has conducted an appraisal to report its results. From high maturity organizations to those just getting started in process improvement, appraisal data helps the SEI to create a more accurate picture of the state of CMM-based process improvement.
Please remember that all appraisals reported to the SEI are treated as proprietary and confidential. At no time will specific individuals, organizations or projects be named in SEI publications or presentations without permission. If the organization prefers to formalize the confidentiality of the appraisal being reported to the SEI, the lead appraiser can use the Non-Disclosure Letter and SEI Non-Disclosure Agreement.
Please support the Lead Appraiser by allowing each appraisal performed at your organization to be reported to the SEI. Lead Appraisers must report appraisal results to the SEI as a condition of their license to perform SCAMPI appraisals. Appraisal performance is a requirement for maintaining Lead Appraiser credentials. Sponsors can request that appraisal results not be reported to the SEI, in which case, the SEI will waive the requirement to submit appraisal results. The Lead Appraiser and team members must know of this condition before committing to perform such appraisals because Lead Appraiser and team members will not accumulate credit toward reauthorization or toward admission to Lead Appraiser training.
return to topWhy should I report my appraisal results to the SEI?
Reporting your organization's appraisal results helps to create a more complete picture of the state of CMMI-based process improvement methods. Understanding the current state of adoption of these methods allows the greater software and systems development community to accomplish the following:
- Examine improvement trends. One example is the Maturity Profile and its interactive version. Neither would be possible without the participation of appraised organizations.
- Improve existing or develop new models. Appraisal data helps to uncover trends in the adoption of an improvement method, often revealing global strengths and weaknesses.
- Support research publications.
- Allow Appraiser training, authorization, and renewal.
Further, if the results of the appraisal are to be announced publicly, used in a formal proposal, or announced outside of the appraised organization, but within the same company, then the appraisal results must be placed on the SEI's Public Appraisal Results Site (PARS).
return to topAre appraisal data kept and verified by the SEI?
SCAMPI Lead Appraisers and SCAMPI B&C Team Leaders are required to submit a data package to the SEI following the completion of SCAMPI appraisals they lead. The SEI uses these data for research and reporting purposes; they are not made public in any way that would compromise the confidentiality of the source.
All data reported to the SEI from appraisals are treated as confidential to the organization submitting the data. This means that access to these data is tightly controlled. Only SEI personnel with a need to know are granted access to identified data.
return to topIf you have a question that is not answered within this FAQ, check the CMMI FAQ Main Page or send email to an SEI person at cmmi-comments@sei.cmu.edu. You will receive a response within 48 hours.