July 3, 2012—An acquisition project starts with the best intentions: field a high-quality system that provides new or improved capabilities to the warfighter. Unfortunately, somewhere between the prototype and deployment of the finished system, the effort goes awry, and the system, as initially envisioned, takes much longer than expected, or may never come to fruition.
As developers spend more time supporting field users, development
progress slows. A lack of either project management experience or a
formal organization also creates problems as developers try to scale the
effort up into a formal program, resulting in a difficult transition as
warfighters wait years for a production-quality system.
The problem is common enough that SEI researchers aggregated work on five independent technology assessments (ITAs) from 2006-2009 to develop a model of this acquisition scenario, titled “The Evolution of a Science Project.” The scenario goes like this: A project begins with an informal team of operational people building a small, throwaway, proof-of-concept prototype to solicit funding. A successful, initial deployment of the prototype builds demand for more capabilities and broader deployment. The project builds on top of the initial prototype to save time, but then starts to find increasing architectural, robustness, performance, usability, and documentation issues. As developers spend more time supporting field users, development progress slows. A lack of either project management experience or a formal organization also creates problems as developers try to scale the effort up into a formal program, resulting in a difficult transition as warfighters wait years for a production-quality system.
Bill Novak, a senior member of the SEI technical staff with the Acquisition Support Program, noted that there are at least two dynamics at work behind the scenes in this scenario. “First, the project is often initiated by operational users who may lack software acquisition expertise,” Novak said. “They focus primarily on quick deployment and operational support, which can result in informal requirements, poor design and code quality, and inadequate documentation.”
The second dynamic is called “firefighting.” “People who are slated to do new development of the next release must be diverted to fix problems in previously fielded releases,” said Novak. “This not only slows new development, but the lack of developers can inject additional problems in the next release that will require even more people to fix.”
Andy Moore, from the Enterprise Threat and Vulnerability Management Team in the CERT Program, described a couple of preliminary findings from the modeling effort. “The accumulation of undiscovered rework from development of the science project prototype moves the development efforts past a tipping point of escalating repair work during the follow-on development.” The tipping point was found to contribute to the “90 Percent Syndrome” experienced in many software development efforts, causing the program to shift from completing normally, to having defects accumulate as rapidly as they can be corrected. Moore commented that “this syndrome occurs during the later stages of development when progress drastically slows down as a much bigger percentage of the effort is devoted to repair work rather than development.”
Some of the key factors behind reaching the tipping point include
“These are preliminary findings, as we will be performing additional model validation in the coming year,” Novak pointed out, while observing that, “the model’s qualitative behavior is similar to what we observe in actual programs.”
The SEI’s work to date on the project will be captured in an upcoming technical report: The Evolution of a Science Project: System Dynamics Modeling of a Recurring Software-Reliant Acquisition Behavior. The team also plans to create an interactive classroom game based on the model to teach better decision making for the defense acquisition workforce that can help lead to better program outcomes. “We’ve gotten feedback from people saying that this kind of hands-on, experiential learning ‘will be incredibly helpful to DoD program offices,’” said Novak.
This article originally appeared in the 2011 SEI Year in Review. To download a copy of the SEI Year in Review, please visit www.sei.cmu.edu/library/abstracts/annualreports/2011-SEI-Year-in-Review.cfm.
For more on this topic, see Bill Novak’s post “The Evolution of Science Projects, Third in a Four-Part Series Exploring Themes Across Acquisition Programs,” in the SEI Blog.
For more information
Please tell us what you
think with this short
(< 5 minute) survey.