Software Engineering Institute

June 15, 2012—"Do more without spending more," urged Ashton Carter in a 2010 memo. Carter was then Under Secretary of Defense for Acquisition, Technology, and Logistics. His call for improved productivity advocated the use of should-cost estimates for major defense programs. Previously, program budgeting depended on estimates that forecasted what a project will cost based on past experience.

Carter called this "business-as-usual management" that essentially required programs to fully expend their budgets. Or, as the SEI's Mike Phillips said, "Once you have that estimate, well, it never seems to come in any cheaper. With the next estimate based on the one preceding it, there is never an opportunity to account for the lessons learned that help software and systems developers get better at their work."

Phillips and Robert Ferguson, both senior members of the technical staff at the SEI who work on the Software Engineering Measurement and Analysis initiative, collaborated on a should-cost analysis of the software used in the F-22 modernization program. The program, which includes upgrades to the aircraft's air-to-ground and intelligence, surveillance, and reconnaissance capabilities, was one of the first United States Air Force programs to use the new should-cost estimation process.

"We aren't just trying to sell a model or a method. We're talking about doing things in a way that has proven to work, and if you don't use the practices, you won't get the performance." - Mike Phillips, SEI

"Our estimate succeeded in finding improvements that could significantly reduce the cost of the program," said Ferguson. To calculate the should-cost estimate, the SEI used the data from the initial basis of the contractor's estimate to create a parametric estimate that closely matched the contractor's. The SEI then used the resulting model to test the sensitivity of the estimate and judge where potential savings could be found and how much could be saved. For example, team performance and estimates of quality could be compared to industry benchmarks. Contractors could then be encouraged to adopt improvements to improve development performance.

One significant source of savings came from improving quality at earlier stages of the lifecycle by adopting best practices. This reduced defects and lowered testing costs. As research has shown—including extensive data from the Team Software Process (TSP) work done at the SEI—repeated testing and defect-removal activities are very inefficient in terms of time and money. "Performance is correlated with using best practices early in the life cycle," says Phillips. "We aren't just trying to sell a model or a method. We're talking about doing things in a way that has proven to work, and if you don't use the practices, you won't get the performance."

The structure of the estimation models assumes that disciplined software development takes less effort than undisciplined development. "This is a reminder to organizations that there is value in bringing discipline to what they do. The estimation tool reflects that value. Estimators have an obligation to show managers the positive effects of high process quality and high levels of development performance. Failure to improve performance could eventually make a contractor non-competitive," Ferguson said.

Another interesting aspect of the F-22 program, according to Phillips, is the plan to focus first on software to increase capability. "People usually want to start with the hardware they want, and then develop software to support it," he said. "Now they can get a lot of capability without having to do hardware upgrades to make it work." While updated hardware is also important to the F-22, rethinking development will enable the F-22 program to get capability improvements sooner. The package of upgrades that included hardware would only have demonstrated benefit in six years; by addressing software first, the estimated time was reduced to two and a half years.

Phillips, a former Air Force test pilot, thinks this shift in thinking is an interesting change. "Think of the way banks have changed," he says. "There used to be lots of paper money going back and forth across the counter. Now the transactions are more and more electronic, and a bank is kind of like a big box with lots of ones and zeros in it. Now a supersonic jet can be thought of more and more as a bunch of ones and zeros with a plane around it."

As the role of software in Department of Defense (DoD) programs has grown, the SEI has brokered compromise between contractors and DoD programs. Both parties must work to improve processes, and they must collaborate on compromises that save time and money without sacrificing quality. Should-cost estimation for software development makes an important contribution to the ultimate goal of making everything more affordable through a collaborative approach between industry and government.

This article originally appeared in the 2011 SEI Year in Review. To download a copy of the SEI Year in Review, please visit http://www.sei.cmu.edu/library/abstracts/annualreports/2011-SEI-Year-in-Review.cfm.