NEWS AT SEI

In August of 2002, the U.S. Army’s acquisition executive, Claude Bolton, asked the SEI to help him with an ambitious, multi-year program to improve the way the Army acquires software-intensive systems. The resultant Army Strategic Software Improvement Program (ASSIP) put into place the necessary infrastructure and initiatives to meet the requirements of section 804 of the FY03 National Defense Authorization Act, which requires defense agencies to develop plans for improving their software acquisitions.

To determine how to improve acquisition processes, the SEI first set about determining the current state of the practice within Army acquisition. Developed specifically for ASSIP, Benchmarking for Improvement, or BFI, is the technique employed. The SEI uses BFI results, along with other data collected for ASSIP, to identify improvement needs across Army acquisition. This article discusses BFI and some of the emerging results from its application on Army programs to date.

Benchmarking for Improvement

The purpose of BFI is to understand the practices used in acquisition programs. Using a model-based question set, a team elicits practices that have been successful in an individual program office as candidate benchmarks for broader Army application. The team also identifies where a program needs practices to help overcome some difficulty (possibly leading to a benchmark for the program). BFI also helps determine where Army policies present barriers to program progress or where absent policies cause ambiguity and increased risk, so that the Army can set its own high-level benchmark targets.

Although BFI applies appraisal techniques, the focus is on discovering what a program office does rather than on measuring its maturity against a model. BFI is not an appraisal, and it does not result in a rating. It is a quick look to identify potential best practices or problems that may be relevant to the program or to the Army in general. However, BFI does retain some familiar appraisal elements: interviews are confidential, and findings require more than one source (human or document) for substantiation. Program managers own the results; to ensure program anonymity, the SEI never publishes attributable information.

To ensure broad system-level coverage, BFI has evolved to include topic areas from the new CMMI Acquisition Module (CMMI-AM). In fact, several BFIs are part of the pilots that will help the Office of the Secretary of Defense refine the CMMI-AM, which is becoming the preferred guide for defense acquisition office improvement.

Emerging Results

Some of the Army representatives who participate in ASSIP suggested that the difficulties in fielding reliable software intensive systems on time and within budget stemmed from not understanding the software aspects of a program. After conducting 9 BFIs during the last 14 months, the SEI has learned that programs are actually struggling with the complexity of the “system of systems” development environment (in which software is only one element) and the challenges associated with managing in that environment. As a result, many recurrent BFI themes have more to do with managing the acquisition process overall.

The table below summarizes some of these key themes, which the SEI will use to help guide Army-wide improvement planning. The Improvement Opportunities section lists issues that need addressing throughout Army acquisition. The Best Practices section lists exemplary responses by individual programs, practices that all Army program offices should consider.

Table 1: Key Themes for Army Improvement Planning

Value to the Programs

For their investment, programs receive immediate and confidential feedback about their current practices, and can leverage that information to develop action plans based on their needs. For example, one benchmarked program used its results to validate its ongoing process improvement initiatives. Another program used its results as the basis for beginning improvement activities. Even some program executive offices (the organization level above the programs) have used BFI results to guide the start of their own improvement work.

Programs also have an opportunity to anonymously critique and influence higher-level policies, with the SEI acting as their advocate. Lastly, each program has a chance to maintain an ongoing relationship with the SEI to provide feedback and gain additional knowledge from other Army programs. Collaboration remains the best method for sharing the findings of the BFIs and other Army work facilitated by the SEI.

The Future of BFIs

As of this writing, the SEI plans to conduct six more BFIs through the end of 2004. Going forward, programs may request BFIs, or they may perform their own self-assessments against the CMMI-AM. As program offices become more experienced with process improvement, the SEI will continue to follow up to determine the successes and shortfalls of the programs’ efforts in order to continually gauge the direction of Army-wide improvement.