As software projects continue to grow larger and more complex, management and control of those projects remains problematic. Cost overruns, late deliveries, and poor-quality products plague developers and acquirers of software-intensive systems in both the defense and commercial sectors. Additionally, the new technologies flooding the market must be evaluated for their benefits and effectiveness. One way to improve this situation is to develop and use measurement techniques and credible information sources for evaluation, benchmarking, and process improvement.

Purpose

The purpose of SEI work in measurement and analysis is to provide analysis guidance, information resources, and practices that help DoD and industry suppliers of software-intensive systems apply quantitative management techniques to improve their projects, processes, and organizations.

Measurement and analysis techniques provide the data organizations need to track efforts to improve software processes, lower costs, reduce defects, and provide valuable information about returns on investments. From introducing basic measurement principles to helping high-maturity organizations introduce statistical process control and improvement methods, the SEI provides the level of guidance necessary to produce meaningful, cost-saving results.

We are extremely pleased with the timely support [the SEI] provides in the areas of measurement and analysis, organizational process performance, and quantitative project management.
Millee Sapp
Software Engineering Process Group Lead,
Warner Robins Air Logistics Center

2002 Accomplishments

Methods Developed to Reconcile CMM, Six Sigma
The SEI, in conjunction with industry collaborators, is leading the way in the application of Six Sigma methods for software. Six Sigma methods are common in organizations that manufacture software-intensive products. Many of these organizations have already implemented software process improvements based on the Capability Maturity Model® (CMM®), but are struggling to align these efforts with manufacturing-based improvement processes such as Six Sigma. In response, the SEI has developed various software process improvement and statistical analysis techniques using Six Sigma methods, an innovation that is yielding significant results. Using these methods, organizations can develop indicators and measures to track customer satisfaction, cost and schedule, defect detection, and other elements of business performance.

During FY2002, these techniques were described by SEI staff members at the International Council on Systems Engineering (INCOSE) conference and coupled with CMMI® process improvement efforts at Warner Robins Air Logistics Center in Georgia. There, the SEI is applying both goal-driven software measurement and Six Sigma methods to help Warner Robins achieve the goals of the Measurement and Analysis and the Quantitative Project Management process areas of the CMMI Product Suite.

Software Engineering Information Repository Reaches 17,000 Members
The exchange of best practices and lessons learned is at the heart of the SEI’s mission. The SEI operates an online resource to disseminate information about software engineering practices and technologies: the Software Engineering Information Repository (SEIR). Software professionals can support their acquisition and development efforts by utilizing this resource for detailed information on a variety of software technologies.

The SEIR is a forum for software engineers from government, industry, and academia to exchange lessons learned, pose questions, and submit materials that might help others to adopt improvement approaches. The SEIR also provides information showing the impact of software engineering improvement methods on organizational performance. Since its inception in 1998 with 104 users and minimal site content, the SEIR has grown to become one of the most frequently visited Web sites operated by the SEI.

One component of the SEIR is the Process Appraisal Information System (PAIS). The PAIS provides the findings and data to support the publication of the Process Maturity Profiles of the Software Community. The maturity profiles characterize the software community in terms of its software process maturity and common process strengths and weaknesses. They are based on results from more than 2,000 process assessments and present information on organization type, size, maturity, and other factors. Nearly 80,000 copies of the maturity profile were downloaded in 2002. The profile is published in March and September every year.

Defect Prediction Model Developed
During FY2002, the SEI and the United Space Alliance collaborated successfully on the topic of capture-recapture (CR) models, developing a prediction model for estimating defects after design or code inspections were conducted on the Space Shuttle. The flight software on the Space Shuttle consists of more than 400,000 lines of code. The completed prediction model combined the CR estimate with size and inspection data to provide a reinspection decision accuracy of 80%—a considerable improvement over previous decision-making accuracy. To date, this has been the largest dataset of actual inspections analyzed with the use of CR models.

Goal-Driven Measurement Techniques Gain Proponents
The SEI’s goal-driven measurement techniques were applied at several Defense Finance and Accounting Service (DFAS) sites. An SEI staff member developed and delivered a measurement workshop to DFAS corporate management in Kansas City, focusing on reusing existing DFAS measurements, rather than starting from scratch. DFAS management is now working independently to develop its own measures and indicators to guide its work. The goal-driven measurement techniques were also taught to practitioners at the Center for Medicare and Medicaid Systems (CMS). Furthermore, these techniques were incorporated into a tutorial on organizational performance measurement that was offered multiple times as part of Carnegie Mellon’s CIO Institute curriculum.