Mary Catherine Ward
Joseph P. Elm
Susan Kushner

Technical Report

CMU/SEI-2006-TR-002
ESC-TR-2006-002

August 2006

Acquisition Support Program

Unlimited distribution subject to the copyright.

[Abstract]  [Acknowledgements] [Executive Summary]    [1 Introduction]   [2 Developing Acquisition Strategies]
[3 Profiling Program Software Characteristics and Acquisition Strategy Drivers]
[4 Key Acquisition Strategy Elements]   [5 Formulating an Acquisition Strategy]   [Conclusion]
[Appendix A: Templates for Software Driver Slider Bars]   [Appendix B: Acronyms]   [Bibliography]
[PDF File]   [Request Form: ASDT Excel File]

Acknowledgements

In developing this report, several individuals made invaluable contributions regarding its content and clarity. Specifically, we want to thank our Carnegie Mellon Software Engineering Institute (SEI) colleagues for their foundational research: Cecilia Albert, John Bergey, Wolfhart Goethert, and Ed Morris. Julie Cohen, Stephen Blanchette, Jr., Brian Gallagher, Mary Popeck, and Eileen Wrubel provided us with insightful feedback. Special thanks go to Thomas Miller for relentlessly testing the functionality and checking the consistency of the Acquisition Strategy Development Tool. Rachel Callison of the SEI Library was instrumental in tracking down elusive reference information. Also, Barbara White of the Communications department demonstrated her Web-savvy by creating the download form for the Acquisition Strategy Development Tool.

Finally, we would like to thank Robert Carnevale of U.S. Army Ground Combat Command and Control for piloting aspects of this research. The support of the U.S. Army Strategic Software Improvement Program (ASSIP) and Dr. James Linnehan of the Office of the Assistant Secretary of the Army for Acquisition, Logistics, and Technology made this work possible.

[Abstract]   [Acknowledgements] [Executive Summary]    [1 Introduction]   [2 Developing Acquisition Strategies]
[3 Profiling Program Software Characteristics and Acquisition Strategy Drivers]
[4 Key Acquisition Strategy Elements]   [5 Formulating an Acquisition Strategy]   [Conclusion]
[Appendix A: Templates for Software Driver Slider Bars]   [Appendix B: Acronyms]   [Bibliography]
[PDF File]   [Request Form: ASDT Excel File]

Executive Summary

As shown in Figure 1, the act of acquisition planning is one of defining and maintaining an overall approach for a program. The acquisition planning process guides all elements of program execution to transform the mission need into a fielded system that is fully supported and delivers the desired capability. The goal of acquisition planning is to provide a roadmap that will be followed to maximize the chances of successfully fielding a system that meets users' needs within cost and on schedule. Acquisition planning is an iterative process; feedback loops impact future acquisition planning activities.

Figure 1: The Context of Acquisition Planning

Many programs struggle during acquisition planning because even though a wealth of information exists, the process itself is not clearly defined or understood. Acquisition planning is a powerful process that can help define the most promising acquisition options that best mitigate risks. In turn, developing an acquisition strategy is a key component of acquisition planning that provides a means of addressing program risks through the program structure. In most cases, however, acquisition planning and acquisition strategy development are done in an ad hoc manner, without consideration of the internal and external factors driving the project. When developing an acquisition strategy, it is important to understand these factors because they often correlate to what drives the program's risk.

For software-intensive systems, the acquisition strategy must acknowledge the type and extent of software risk to the program and include plans to actively mitigate those risks. Programs need structured ways to reason about software risks, formulate acquisition strategies to mitigate software risk, and evaluate their current acquisition strategy in an ongoing, systematic manner.

The acquisition strategy guides decisions made throughout the life cycle of the program. It should be based on reasoning about both internal and external complex factors with regard to the system. When developing an acquisition strategy, it is important to understand the program's driving factors. These drivers are conditions that impact program risk. For example, the skills of the personnel working in the program office, the number of stakeholders, and the number of system configurations that need to be maintained are all drivers. Drivers can influence the acquisition strategy independently or there can be interactions between drivers that affect program risk. For example, unstable requirements usually influence overall costs.

Figure 2 shows the relationships among the program's drivers, its risks, and its acquisition strategy.

Figure 2: Relationships Among Drivers, Risks, and Acquisition Strategy

The research results presented in this report support a more systematic approach to reasoning about software risk on a program. The methods and techniques presented contribute to the work that focuses on developing an acquisition strategy from a sound, systems engineering approach.

This report outlines research performed to help acquisition planners more systematically

profile conditions that impact program risk (drivers)

identify sources of potential software risk early in the program and throughout the program life cycle

develop an acquisition strategy by decomposing it into the strategy elements that compose it, addressing them individually and then collectively

reason about their acquisition strategy's ability to mitigate software risk in an ongoing manner

The report introduces a taxonomy of strategy drivers and strategy elements. It also provides a method for performing a comparative analysis of the strategy drivers and the resulting strategic choices for the elements. For a program with an acquisition strategy already in execution, the method can be used to perform a strategy analysis. The method uses a technique, slider bars, to support a more systematic approach to reasoning about software risk on a program.

For example, to use this method and the slider bar technique to develop a strategy, acquisition planners would perform the following steps:

Define the objectives of the acquisition.

Identify and evaluate the factors that drive the program.

Decompose the strategy into individual strategy elements.

For the selected strategy elements, identify the potential strategic choices and rank them in order of their risk mitigation capabilities for your program.

Evaluate the strategy drivers for the program to identify those that influence the strategy element through the introduction of risk that may be mitigated by the strategy element.

Define the relationship between the risk generated by the strategy driver and the risk mitigation capabilities of the strategy element.

Map the driver evaluations from Step 2 to the strategy element using the slider bars.

Choose the strategy that best mitigates the risk elements.

Identify residual risks.

The Acquisition Strategy Development Tool (ASDT) is a customized Excel workbook we created to help acquisition planners work through the method and techniques. The ASDT is provided so that acquisition organizations do not have to develop slider bar templates from scratch. The ASDT is available for download on the SEI Publications Web site. Note that acquisition planners can also apply the method and techniques discussed in this report without using the ASDT.

Our research focused on identifying and mitigating software risk in a program during the acquisition planning process. However, the methods and techniques we present can be applied to acquisition planning areas other than software risk.

1 Introduction

1.1 Purpose

The goal of acquisition planning is to provide a roadmap that will be followed to maximize the chances of successfully fielding a system that meets users' needs within cost and schedule. An acquisition plan guides all elements of program execution to transform the mission need into a fully supported, fielded system that delivers the desired capability.

Many programs struggle during acquisition planning because even though a wealth of information exists, the process itself is not clearly defined or understood. Software acquisition planning is a powerful process that can help define the most promising acquisition options that best mitigate risks in the development of software. Developing an acquisition strategy is a key component of acquisition planning that provides a means of addressing program risks through the program structure. In most cases, however, acquisition planning and acquisition strategy development is done in an ad hoc manner, without consideration of the internal and external factors driving the project. When developing an acquisition strategy, it is important to understand these factors because they often correlate to what drives the program's risk.

During the past year, the Acquisition Support Program at the Carnegie Mellon Software Engineering Institute researched more systematic approaches to reason about software risk on a program. Specifically, we explored answers to the questions "Are the methods and techniques arising from a sound, systems engineering approach ones that can be applied to acquisition planning and acquisition strategy development? Will they enable programs to better reason about software risks, formulate acquisition strategies to mitigate those risks, and evaluate their current strategy in an ongoing, systematic manner?"

This report outlines the research that was performed and is designed to help acquisition planners more systematically

profile conditions that impact program risk (drivers)

identify sources of potential software risk early in the program and throughout the program life cycle

develop an acquisition strategy by decomposing it into the strategy elements that compose it, addressing them individually and then collectively

reason about their acquisition strategy's ability to mitigate software risk in an ongoing manner

The report introduces a taxonomy of strategy drivers and strategy elements. It also provides a method for performing a comparative analysis of the strategy drivers and the resulting strategic choices for the elements. The method proposed is a bi-directional, graphical method of examining and analyzing the relationships between the drivers and the strategic choices. The method is bi-directional in that it enables a program to

analyze strategy drivers and choose strategies for each strategy element to minimize the risks induced by those drivers, or

choose a strategy for each strategy element and analyze the risks induced by the program's strategy drivers

The method uses a technique, slider bars, to support a more systematic approach to reasoning about software risk on a program. This work extends a concept introduced by the Department of the Navy representing strategy elements as a continuum or sliding scale, much in the way the volume control on a stereo can range from soft to loud [Department of the Navy 01]. Slider bars can be used to graphically visualize a program's strategy drivers, acquisition strategy element strategic choices, and the relationships among the drivers and choices. The Acquisition Strategy Development Tool (ASDT) explained in Section 1.3 is provided to assist program offices work through the methods and techniques.

1.2 Document Structure

Section 2 discusses the need for acquisition planning and the roles of the acquisition strategy and acquisition plan in the acquisition planning process. It defines key terms and highlights the relationship between acquisition planning and software acquisition planning including the challenges posed by software.

Section 3 introduces a taxonomy of strategy drivers to help programs identify, evaluate, and understand acquisition strategy drivers. It discusses each strategy driver to assist program personnel to assess their program's software acquisition risk.

Section 4 discusses acquisition strategies in more detail and sample acquisition strategy elements including Acquisition Approach, Competition, Solicitation, Contract Approach, Training, and Source of Support. This is not a comprehensive set of strategy elements; it is only a subset used to illustrate the acquisition strategy development process proposed in this report.

Section 5 provides a method for performing a comparative analysis of the strategy drivers and the strategic choices for each strategy element. The method uses a technique, slider bars to support a more systematic approach to reasoning about software risk on a program. This section describes how to profile a program's strategy drivers, its acquisition strategy elements and corresponding strategic choices, and the relationship between the drivers and strategic choices for each element.

Appendix A provides a hard copy of the template used in the ASDT. This tool is discussed further in Section 1.3. The template can be used to profile the program's key strategy drivers. It can also be used to identify specific strategy element choices and evaluate how those choices mitigate the program's software risks.

1.3 Acquisition Strategy Development Tool

The purpose of the ASDT is to help government program offices formulate and evaluate how their acquisition strategies address the major software risks faced by the program in a more systematic way. It can be used to profile the program's software acquisition characteristics and identify key strategy drivers. It can also be used to identify specific strategic choices and evaluate how those choices mitigate the program's software risks.

Acquisition planners can apply the method and techniques discussed in this report without using the ASDT. The ASDT is provided for acquisition planners so that each acquisition organization does not have to design templates from scratch. The ASDT is available for download on the SEI Publications Web site.

Instructions on how to use the ASDT are contained within the workbook. The ASDT requires Microsoft Office Excel 2003 or newer. Macro execution must be enabled for it to work properly.

2 Developing Acquisition Strategies

2.1 Acquisition Planning Includes an Acquisition Strategy and an Acquisition Plan

Acquisition planning is defined as

The process by which the efforts of all personnel responsible for an acquisition are coordinated and integrated through a comprehensive plan for fulfilling the agency need in a timely manner and at a reasonable cost. It is performed throughout the life cycle and includes developing an overall acquisition strategy for managing the acquisition and a written Acquisition Plan (AP) [DAU 03].

As shown in Figure 3, acquisition planning is the act of defining and maintaining an overall approach for a program. Acquisition planning guides all elements of program execution to transform the mission need into a fielded system that is fully supported and delivers the desired capability. The goal of acquisition planning is to provide a roadmap that will be followed to maximize the chances of successfully fielding a system that meets users' needs within cost and schedule. Acquisition planning is an iterative process; feedback loops impact future acquisition planning activities.

Figure 3: The Acquisition Planning Context

Software acquisition planning is a powerful process that can help define the most promising acquisition options that best mitigate risks in the development of software. Program risks take many forms and come from many sources. The first challenge to the program management team is to identify all of the affected stakeholders. The second challenge is to apply the expertise of program office staff and these stakeholders to identify and prioritize the program's goals and risks. After these two challenges are met, it is possible to reason about and create an acquisition strategy that mitigates the highest priority risks and manages the remaining risks.

An acquisition strategy, when formulated carefully, can be a means of addressing program risks via program structure. More formally, an acquisition strategy is

A business and technical management approach designed to achieve program objectives within the resource constraints imposed. It is the framework for planning, directing, contracting for, and managing a program. It provides a master schedule for research, development, test, production, fielding, modification, postproduction management, and other activities essential for program success. The acquisition strategy is the basis for formulating functional plans and strategies (e.g., Test and Evaluation Master Plan (TEMP), Acquisition Plan (AP), competition, systems engineering, etc.) [DAU 03].

The "best" acquisition strategy for a given program directly addresses that program's highest priority risks. High-priority risks can be technical, if no one has yet built a component that meets some critical aspect of the system or has never combined mature components in the way that is required. Risks can be programmatic if the system must be designed to accommodate predefined cost or schedule constraints. Or, risks can be mission-related when the characteristics of a system that meets the need cannot be fully articulated and agreed upon by stakeholders. Each program faces a unique set of risks, so the corresponding acquisition strategy must be unique to address them.

In addition, program risks change over the course of the program as new risks are discovered and previously identified risks are mitigated. The selected acquisition strategy must evolve to respond to what is known (and unknown) about the goals, objectives, and constraints of the system; the state of the requirements; the maturity of the technology, including the software; and the available resources such as budget, people, and skills. Much as a commander creates and adjusts a course of action based on a sense of the threat and the operational constraints and resources available to meet that threat, a program manager uses acquisition planning to continually adjust the program's acquisition strategy to accommodate the program's risks as they are currently understood.

Figure 4 represents a high-level graphical view of acquisition strategy evolution. First, an initial strategy is developed and executed. Then, the acquisition strategy is refined based on progress and modifying forces to the program. This cycle is repeated multiple times until the program completes.

Figure 4: The Acquisition Strategy Development Process

As stated previously, an acquisition strategy is one of two major components of acquisition planning. The second necessary component is a written acquisition plan. An acquisition plan is a formal written document reflecting the specific actions necessary to execute the approach established in the approved acquisition strategy and guiding contractual implementation [DAU 03]

The terms acquisition planning, acquisition strategy, and acquisition plan are frequently used interchangeably, which causes much confusion. A program's acquisition strategy is different from its acquisition plan, but both are artifacts of the process of acquisition planning.

2.2 Key Aspects of Acquisition Strategy Development

An acquisition strategy must take into consideration many elements to accomplish system objectives during the course of the system's life cycle [DoD 96]. An acquisition strategy includes a collection of strategy elements. Each strategy element is a decision or a plan on how to proceed with a specific aspect of the program execution. Examples of strategy elements include, but are not limited to, the acquisition approach, contract approach, and competition. Section 4 describes strategy elements in more detail.

The acquisition strategy guides decisions made throughout the life of the program. It should be based on reasoning about both internal and external complex factors with regard to the system being built, operated, and maintained. When developing an acquisition strategy, it is important to understand the program's driving factors. Drivers are conditions that impact program risk. For example, the skills of the program office personnel, the number of stakeholders, and the number of system configurations that need to be maintained are all drivers. Drivers can influence the acquisition strategy independently or there can be interactions between drivers that affect program risk. For example, unstable requirements usually influence overall costs. Section 3 contains a taxonomy to help acquisition planners profile the program and determine the extent to which it is exposed to software risk.

Figure 5 shows the relationships among the program's drivers, its risks, and its acquisition strategy. The first step is to determine a program's internal and external drivers.

After identifying the major drivers, you need to

determine which drivers present the highest risk to your program

determine how the drivers will influence your program's acquisition strategy elements

formulate strategies (by making choices among alternatives) that you believe will best address the highest risk drivers

analyze the strategies to determine gaps and remaining high risk areas and determine if these risks can be mitigated through other options

No strategy can mitigate all of a program's risk; identify the strategy that provides the best risk mitigation and can be executed by the available staff and within the available budget.

Figure 5: Risk Management Via Acquisition Strategy

2.3 Software Risk in Acquisition Planning

Acquisition planning must address the risks that can arise during development, use, and maintenance of software. To this end, the acquisition strategy must reflect

the degree to which software determines the capabilities and qualities of each system component or sub-component

all software elements that are potentially difficult or costly to implement

the operational, maintenance, and sustainment implications of software components

the threat that faulty or failed software poses to human life, safety, security, environment, and so on

Some acquisition planners neglect to address the unique nature of software risks. They ignore them or treat them as an insignificant part of the system implementation risks and assume that they will be managed by the implementation contractor. This paradigm persists for a variety of reasons.

The acquisition community as a whole does not address software early enough in system development. Therefore, many critical acquisition planning decisions are made with scant consideration of software.

Acquisition planners have no empirical foundation for selecting one acquisition approach over another. This makes it difficult to know whether a given approach can successfully mitigate software risks and encourages managers to fall back on the approaches with which they are most familiar; these approaches usually ignore software risks.

2.3.1 The Challenge of Software

Increasingly, software is the major determinant of system functionality. In the year 2000, the Defense Science Board found tremendous growth in the software content of both manned and unmanned systems [DSB 00]. In fact, software requirements now represent the bulk of overall specification requirements for most systems. For example, 80% of the U.S. Air Force's requirements for building the F/A-22 Raptor and 65% of the requirements for the B-2 Spirit Stealth Bomber were software requirements [Nelson 99].

Within the U.S. Army, the growing dependence on software is equally astounding. The M1 Abrams Main Battle Tank (MBT) performs its mission with the help of approximately 600,000 (0.6 M) source lines of software code (SLOC) [Nelson 99]. Planned Army systems such as those under development by the Future Combat Systems network employ far more software (34M SLOC) [GAO 05].

Figure 6: A Survey of Software Projects According to the Standish Group

Figure 6: A Survey of Software Projects According to the Standish Group

Due to the advanced capabilities now required, software increasingly drives the cost, schedule, and quality of integrated, hybrid (software and hardware) systems. As shown in Figure 6, the percentage of software projects completed on time seems to be improving. However, in 2002, 51% of software projects were late, over budget, and/or lacked critical features--and this number excludes an additional 15% of projects that were cancelled. In addition, the average final software product contains only 52% of the originally specified features [Standish 03].

Systemic analysis of results from the Tri-Services Assessment Initiative suggest that there has been "little positive impact from past corrective actions, initiatives, and policy" in the Department of Defense (DoD) [McGarry 03]. There are many reasons why software, particularly DoD software, has proven so difficult to build. We cannot possibly list them all in this document, but we can reflect on several reasons that illustrate why formulating an acquisition strategy is critical.

It is likely that no organization acquires a wider range of software than the DoD. In the Army alone, software ranges from well-defined business systems to systems supporting the acquiring, tracking, and destroying of a missile moving faster than a bullet. The broad range of software required to support these two different missions implies that an equally broad range of strategies, processes, and technologies must be employed.

Some of the most complex types of software produced by the DoD are components embedded inside large combat systems. In many of these systems, software is an enabler that, when used in combination with other technologies, can be used to address system/subsystem requirements. In many cases, alternative technologies could be used to provide the same functionality--albeit with different implications. The complexity of the software required is rarely evident when the system is conceived and the software requirements tend to grow in complexity as hardware component builders make assumptions about specific systems and the community of interacting systems. Eventually, software must be developed to fill the gaps and to integrate individual, custom systems.

Engineers in other disciplines understand the limitations of the materials and associated fabrication techniques within their discipline. For example, there are rating scales for properties of metals such as thermal stability, tensile strength, hardness, elasticity, and so on. On the other hand, there are few, if any, hard and fast "rules" of software engineering. In this environment, planners, developers, stakeholders, and users all tend to make optimistic assumptions about software--that it is infinitely malleable and can be made to exhibit unprecedented degrees of interesting characteristics (e.g., dependability, performance, and security). In effect, flexibility is the greatest asset of software but is also its key liability.

The DoD acquisition community is rightly focused on acquiring capabilities to support the warfighter. However, this tight focus sometimes leads to reduced emphasis on the problems of developing appropriate software to support these capabilities because the engineering of the software becomes secondary to the engineering of the hardware for the system. Because software is inherently intangible and often a relatively small part of the system, other system issues overwhelm it. As a result, software development is not properly planned, started late, defined late, under-budgeted, and so on. Ultimately, a software implementation that might have been straightforward at one time stagnates until there is no option but to attempt to develop a now-complex software component on a shortened schedule.

Effective software acquisition can be a point of great leverage. Addressing even some of the problems with software in major system acquisitions could lead to huge savings and better systems. The DoD has recognized this for many years and has attempted to improve the situation by instituting various standards such as DoD-STD-2167A, MIL-STD-498, IEEE/EIA 1207, by embracing process improvement methodologies such as ISO 9000 and the Capability Maturity Model Integration (CMMI) framework, and by adopting a wide range of newer, more flexible technologies such as Extensible Markup Language (XML). However, in almost all cases, as evidenced by the Tri-Services Assessment Initiative, results have been disappointing [McGarry 03].

2.3.2 Planning to Mitigate Software Risk

Software acquisition planning is a powerful process that can help define the most promising acquisition options that best mitigate risks in the development of software. An acquisition strategy should clearly describe the program's approach for defining, building, fielding, and supporting the system by

documenting what the program understands to be the need and the environmental constraints under which the system must be built, operated, and maintained

defining the means by which requirements that reflect the need will be developed and managed

establishing appropriate relationships and commitments among stakeholders, program offices, and contractors for the life of the system

providing a framework under which a contract (or contracts) is managed to deliver products or services needed by the program

establishing mechanisms to effectively communicate the approach that is used by the program to all affected stakeholders

A software acquisition strategy must effectively communicate a complete, coherent, and consistent approach that is used by the program so that the program can be resourced, staffed, and tracked. Above all, the software acquisition strategy must support the DOD directive to

acquire quality products that satisfy user needs with measurable improvements to mission capability and operational support, in a timely manner, and at a fair and reasonable price [DoD 03a]

Software acquisition planning aimed at mitigating software risk begins as soon as it becomes apparent that a materiel solution employing software might be required--or in the earliest days of Concept Refinement for a program that uses all of the acquisition phases described in DoD Instruction Operation of the Defense Acquisition System (DoDI 5000.2) [DoD 03b]. At no other point is the program as pliable. A software acquisition strategy is custom-built to help the system s1takeholders, program management office¹ (PMO), and contractors achieve the best software system possible within program constraints. While these early steps in acquisition planning are the first and arguably the foremost mechanism available to achieve objectives in the acquisition and sustainment of systems, the acquisition strategy must continue to evolve as concepts are refined, as technology is developed, as systems are acquired, and as systems are fielded and supported--until the systems are retired.

3 Profiling Program Software Characteristics and Acquisition Strategy Drivers

This section introduces a taxonomy of categories and drivers, depicted in Figure 7, to help a program determine the extent to which it is exposed to software risk. The categories and corresponding drivers are designed to be used by acquisition planners to profile software risk early in the program. The software risk profile should be updated throughout the execution of the program and the acquisition strategy should be adjusted to address newly discovered risks and risks that have been mitigated.


Software Criticality Category	Acquisition Environment Category	Programmatic Category	Organizational Category	Life Cycle Category

Software Criticality	Policies and Mandates	Mission Needs and Scope	PMO Capability	Product Definition and Specification
	Supplier Availability	Funding	Stakeholders	Architecture and Design
		Schedule	Supplier Capability	Verification and Test
				Deployment
				Maintenance and Support
				Disposal

Figure 7: Categories of Program Software Characteristics and Subsequent Strategy Drivers

As shown in Figure 7, program characteristics and strategy drivers are aggregated into five main categories:

Software criticality quantifies the extent to which the system is "software-bound." Systems are considered software-bound when they require large amounts of software, highly complex software, or software that is directly responsible for fulfilling critical aspects of the system mission.
Acquisition environment program characteristics and associated strategy drivers quantify the extent to which the environment in which the acquisition occurs affects the software risk in the program. The level of acquisition environment risk can be represented in terms of policies and mandates imposed on the program and the availability of suppliers.
Programmatic program characteristics and associated strategy drivers quantify the extent to which the fundamental programmatic aspects of scope, funding, and schedule affect the software risk in the program.
Organizational program characteristics and associated strategy drivers quantify the extent to which organizational factors inside and outside the program affect the software risk in the program. The level of organizational software risk can be represented in terms of the characteristics of the program management office, supplier, and stakeholders.
Life-cycle program characteristics and associated strategy drivers quantify the extent to which various life cycle facets pose risk to the program. The level of life-cycle software risk can be represented in terms of the risks associated with the definition, specification, architecture, design, implementation, test, deployment, operations and maintenance, and disposal of the system.

Each of these categories, subsequent strategy drivers, and sublevel drivers is discussed in more detail in the following sections. One or more of the drivers may be difficult to determine or not applicable during a particular program time frame.

3.1 Software Criticality Category

3.1.1 Software Criticality Driver

The extent to which an individual program must specifically focus on and address software risks depends on the criticality of software within the capability to be deployed. To determine software criticality proportion and reliance need to be evaluated.

Proportion is the ratio of the software to the entire system and is usually evaluated using life-cycle cost, time needed to produce and maintain the software, and functionality provided. For example, is software the majority of the system or a minor part of the system? For certain categories of business systems, such as financial, human resources, inventory management systems, and for other categories of systems that support the mission of the DoD such as Command, Control, Communications, Computers, Intelligence and Reconnaissance systems, the proportion of the software to the system is obvious. For these types of systems, software virtually is the system.

For other types of systems, however, the significance of software components is not as obvious. In these systems, the software may not be configured as a set of large applications that appear on a software architecture diagram at the system level, but instead, software exists in the system as relatively small chunks distributed across a number of other systems or devices. In this case, the relationship between the software components and hardware components may be more dominant than the relationship between the individual pieces of software. The proportion of software might be small, but the degree to which the important functionality of the system depends on software is large. That is why an evaluation of reliance is necessary to determine software criticality.

Reliance is the degree to which the important functionality of the system depends on the software. For example, a system running flight control software that is needed to fly a helicopter has a higher reliance on software than a system that employs software as a means of modeling certain system characteristics.

What can be deceiving about the criticality of the software to be procured, however, is that in many modern systems, evaluating proportion alone is insufficient and the reliance the system has on the software must also be evaluated. For example, modern cars--like their 1909 Model T counterparts--are systems that rely on a drive train to impel the wheels, a steering linkage to control the direction of travel, and brakes to stop them. Unlike the Model T, however, virtually no modern car can perform in more than a "limp home" mode without software to control many mechanical functions.

The software used by the 2003 BMW 745Li, shown in Figure 8, is a good illustration of the increasing software control of hardware systems in automobiles. Among the many features that are likely to rely on software are the "mayday" phone, on-board navigation system, dynamic stability control, dynamic traction control, active roll stabilization, dynamic brake control, coded drive-away protection, an adaptive automatic transmission, and iDrive systems. This list can be extended to include the software that controls engine performance, emissions, and fuel consumption found on virtually all modern cars.

Figure 8: BMW 745Li Software

The trend toward increased software control can also be seen in U.S. military systems. For example, in the 1975 version of the F-15 Eagle only 35% of functions required software support. That number grew to 80% for the F/A-22 Raptor introduced 20 years later [Nelson 99].

One definition of the criticality of the software in a system can be found in Table 1.

Table 1: Relationships Among Criticality, Reliance, and Proportion

Criticality	Reliance	Proportion
Very High	Complete reliance on software	Majority of the system
High	High reliance on software	Majority of the System
Medium	High reliance on software	Minor part of the system
Low	Low reliance on software	Minor part of the system
Very Low	No reliance on software	Software is not part of the system

The software in a system (or software used to model, simulate, or test a system) cannot be categorized simply as critical or not. Most programs categorize the software somewhere between the two extremes. The extent of the software criticality can be rated on a sliding scale that extends from one extreme to the other: Very Low to Very High.

If software criticality is low, software risks are not dominant factors in acquisition planning. However, systems that are not software-critical are increasingly rare. Therefore, most programs today need a way to reason about software risks and the potential for the chosen acquisition strategy to mitigate software risk within the system.

3.2 Acquisition Environment Category

Evaluating acquisition environment program characteristics and associated strategy drivers, including policies and mandates and supplier availability, can help quantify the type and amount of software risk in the program.

3.2.1 Policies and Mandates Driver

Policies and mandates are external constraints that a higher authority imposes on a program. One definition of policy is "a definite course or method of action selected from among alternatives and in light of given conditions to guide and determine present and future decisions" and another is "a high-level overall plan embracing the general goals and acceptable procedures especially of a governmental body." The dictionary also defines a mandate as "an authoritative command" that a higher authority imposes [Merriam-Webster 05].

Policies and mandates come in several forms and can be described by the examples shown in Table 2.

Table 2: Examples of Policies and Mandates


Policy or Mandate	Example
Compliance with one or more laws	OSHA Privacy Act
Compliance with one or more directives or instructions	Command, Control, Communications, Computers, Intelligence, Surveillance and Reconnaissance (C4ISR) Business Enterprise Architecture (BEA) Joint Technical Architecture (JTA) DoD 8500 series on Information Assurance
Compliance with a specific practice	Use of performance requirements instead of detailed specifications Use of a Statement of Objective (SOO) instead of a Statement of Work (SOW) in the initial solicitation Use of a Capabilities Development Document (CDD) instead of an Operational Requirements Document (ORD)
Compliance with specific contracting elements	8A requirements
Required certifications and accreditations	Certificate of Networthiness (CoN) Certificate to Operation (CtO)
Use of Government Furnished Information (GFI) and Equipment (GFE)	Incorporation of a specific component Providing the contractor with a development and testing environment

When formulating an acquisition strategy, acquisition planners must consider the amount of conflict between the policies and mandates imposed on a program, the relative stability or instability of a particular policy or mandate, the degree to which policies and mandates conflict with program needs, and the amount of control a program has to modify a policy or mandate.

Analyzing the applicability of mandates, resolving any conflicts, and determining their impact, regardless of their relevance to the program, requires resources. Traversing the paper trail to verify compliance with policies and mandates can be quite time-consuming; these tasks should not be underestimated during planning. Information about regulations abounds, but gathering information about every single policy and mandate that applies to a program is a nontrivial task. For example

Who is responsible for figuring out the detail of compliance: the PMO or the contractor?

What is the risk something will be overlooked?

What is the risk that a policy or mandate can be interpreted in multiple ways?

The more policies and mandates imposed on a program, the more resources consumed, and more likely that one policy or mandate will conflict with another policy or mandate. Also, after a conflict between mandates is identified, there may or may not be a consistent mechanism to resolve the conflict and programs confronted with such conflicts rarely receive guidance on how to resolve them. The PMO may assume some risk when the contractor resolves a conflict that the PMO doesn't know about or if the PMO does not fully understand what the resolution means. Whether the PMO or contractor attempts to resolve a conflict between mandates, there is a risk that the resolution can cause technical incompatibilities with other systems in the future or inefficiencies in the current system.

In addition to conflicts arising between individual policies and mandates, conflicts between the policies and mandates and the needs of the program can also arise. These types of conflicts can be manifested in many forms including

expense of conforming to policies and mandates without having additional funding to do so. For example, a requirement to use a specific software component might be imposed, but funding for the sustainment of that component is not provided or guaranteed.

requirements that specific practices be followed (or artifacts created) that are untried and unproven and have not been adequately piloted before being imposed. For example, one program producing real-time embedded software was directed to develop all code in Java at a time when Java did not support real-time software.

expense of significant rework and schedule degradation when policies or mandates are levied changes after the system has already been designed

requirements to use a solution developed by, or for, the DoD that competes with more viable commercial alternatives

schedule disruption caused by mandates. For example, meeting National Security Agency certification requirements can become a critical path in the program schedule and must be planned.

work to determine which policies and mandates apply to your program and what the impacts are; in a large program, determining how the policies and mandates can be applied consistently to all facets of the program. Even allocating the proper staff to monitor these issues can be difficult for a PMO.

As in the case of conflicts between policies and mandates, after a conflict is identified between policies and mandates and program needs, there may or may not be a consistent mechanism to resolve the conflict. In addition, alternative solutions may not be allowed.

The stability or instability of a policy or mandate must also be considered. Is there a risk that the program will adhere to a policy or mandate that will change after key program decisions have been made? What is the likelihood that a policy or mandate will be introduced midway through the program that would require the solution to be reworked?

A program's ability to resolve policy and mandate conflicts may counterbalance the risk imposed by the number and extent of the conflicts. It is important to have a good understanding of which conflicts are beyond PMO resolution so that appropriate higher level help can be sought. The presence of too many irresolvable conflicts can be a warning sign that the program cannot be executed as currently planned.

The extent to which a program has policy and mandate conflicts can be rated on a sliding scale that extends from one extreme to the other: Low to High.

3.2.2 Supplier Availability Driver

Another external environmental factor to consider when evaluating an acquisition strategy is the number of available, qualified suppliers of required software components. How many suppliers can potentially fulfill the needs of the program? Is the program constrained by a limited set of pre-qualified suppliers? Does their expertise match the needs of the program? Are there unique aspects of the program that can only be supported by a limited number of suppliers? Do the suppliers have adequate capacity to execute all of the expected work within the required time frame? The number of available, qualified suppliers can be rated on a sliding scale that extends from one extreme to the other: Low to High.

3.3 Programmatic Category

Programmatic program characteristics quantify the extent to which fundamental programmatic strategy drivers of mission needs and scope, funding, and schedule affect the type and amount of software risk in the program.

3.3.1 Mission Needs and Scope Driver

Systems are developed to fulfill the needs of the mission. Every acquisition plans to fulfill all defined mission needs and deliver a product that satisfies 100% of the user requirements. And yet we see from the Standish Group's CHAOS report that "successful" systems meet only 52% of requirements [Standish 03]. Clearly, the mission needs contain some degree of flexibility. The flexibility of the mission that the system needs to fulfill is a strategy driver. For some missions, the scope of the mission is very exact and constricted. For others, it is much more flexible. The extent to which the mission needs are constrained can be rated on a sliding scale that extends from one extreme to the other: Flexible to Rigid.

3.3.2 Funding Drivers

The funding drivers described in this section include funding constraints and the funding profile.

3.3.2.1 Funding Constraints

Funding constraints determine the degree to which the amount of funding allocated to the program matches the estimated funding required to develop, operate, and maintain the system. Funding constraints can significantly impact both "software-critical" and "non-software-critical" systems. Inadequate funding can cause suboptimal engineering choices to be made in the interest of saving money; choices such as deferring functionality or allocating requirements to software that might be better implemented in hardware.

How does software influence the risk associated with funding constraints? A key consideration is the life-cycle cost and total cost of ownership estimates for the software in the system. According to DoDI 5000.2, sound cost estimates are based on well-defined programs. With large-scale unprecedented systems, this becomes a difficult task.

The inability to develop credible and accurate cost estimates correlates with the level of uncertainty in the program or system definition, technical performance, and cost estimating method. In turn, this influences the accuracy of the anticipated funding profile. In contrast, the funding profile is sometimes dictated by the congressional budget. The program must then use its cost estimating ability to determine an affordable set of requirements.

Factors influencing cost estimating include immature software cost estimation models, lack of experience in using cost models, lack of knowledge and experience in planning and managing software development programs, lack of subject matter experts, immature and volatile requirements, the quality of historical data from similar projects, and influences by executives having the authority to reject or modify valid estimates due to "political" or other reasons.

The extent to which the funding constraints are a fact of the acquisition can be rated on a sliding scale that extends from one extreme to the other: Few to Many.

3.3.2.2 Funding Profile

The type and timing of funding dollars applied to the acquisition can also drive decisions made about the acquisition strategy. It is not just how much funding the program receives, but also the type of appropriation and when it is received. For example, a program that needs to do a lot for research and development and will acquire a limited amount of hardware usually has a front-loaded budget, in which a majority of the overall funding is allocated to do the work at the beginning of the project versus the end. In addition, this program would need research and development dollars.

In order to develop high levels of product quality, investment in up-front engineering tasks is necessary. Tasks such as requirements definition and architecture drive the scope of the software development effort, so inadequate attention to those tasks invariably leads to increases in software size and complexity and ultimately to schedule delays, cost increases, and product quality issues. Consequently, early engineering tasks need to be funded adequately at the outset of the program.

Using an evolutionary acquisition approach such as incremental or spiral can complicate funding needs. For example, the program may need to budget funding from several different appropriation types simultaneously.

The extent to which the funding profile matches the needs of the acquisition can be rated on a sliding scale that extends from one extreme to the other: Mismatched to Matched.

3.3.3 Schedule Drivers

The schedule drivers described in this section include schedule constraints and urgency.

3.3.3.1 Schedule Constraints

Like funding constraints, schedule constraints are the degree to which the schedule allocated to the program matches the estimated time required to develop, operate, and maintain the system. Schedule constraints can significantly impact both "software-critical" and "non-software-critical" systems.

How does software influence the schedule for the system? What aspects of the program have key dependencies on certain software events? If software is on the critical path, software risks need to have very good mitigation plans and must be constantly monitored.

The ability to develop credible and accurate schedule estimates is influenced by the level of uncertainty in the program or system definition and the schedule estimating method used. Other factors that influence schedule estimating include immature estimation models, lack of experience in using the models, lack of knowledge and experience in planning and managing software development programs, lack of subject matter experts, immature and volatile requirements, the experience of the personnel performing the estimates and the quality of historical information from similar projects, and influences by executives having the authority to reject or modify valid estimates due to "political" or other reasons.

Another consideration that can have a large impact on schedule risk is the need to interface with other programs. Risk is increased if, due to interoperability goals, the program schedule must be coordinated with one or more other programs.

The extent to which schedule constraints are a fact of the acquisition can be represented as a continuum that extends from one extreme to the other: Few to Many.

3.3.3.2 Urgency

In certain circumstances, urgency of deploying a capability to the field might have significant impact on the acquisition strategy. For example, the current war situation might be placing pressure on certain acquisitions to deliver capabilities at a different rate, in a different sequence, or with different capabilities than planned. How does your acquisition strategy accommodate urgent needs? The level of urgency can be rated on a sliding scale that extends from one extreme to the other: Low to High.

3.4 Organizational Category

Organizational program characteristics and associated strategy drivers quantify the extent to which organizational factors inside and outside the program affect the software risk in the program. Strategy drivers include PMO capabilities, stakeholders, policies/mandates, and performance team/contractor capabilities.

3.4.1 PMO Capabilities Drivers

The drivers pertaining to PMO capabilities described in this section include staff skills, capacity and stability, and the process focus of the PMO and its governing organization.

3.4.1.1 PMO Staff Skills

The PM [program manager] must assemble the proper acquisition strategy selection and development team. It is important to staff this team with individuals whose knowledge, experience and access to pertinent information equips them to effectively address all of the topics. The success of each of the succeeding steps in the selection process depends on the active participation of all the members of the team. Good contracting, technical and business/financial managers will be key payers in the selection of the acquisition strategy [Department of the Navy 01].

With respect to having the proper software expertise, the GAO noted: "The more managers know about software development processes and metrics, the better equipped they are to acquire software" [GAO 04a]. In effect, the government program manager needs to be a "smart buyer" of the software within the system.

While some level of software expertise is necessary, it is not the only ingredient in the recipe for success. Systems engineering is also a critical skill and an inherent PMO responsibility. Therefore, the PMO must also have adequate systems engineering staff. In addition, for software critical systems expertise is needed in all areas including software management (cost and schedule estimating, metrics definition and analysis, process improvement), software acquisition, software architecture, software and systems integration, software development (including commercial off-the-shelf [COTS], reuse, and open systems), security and accreditation and organizational change management.

The expertise needs to be brought on early in the program to develop the acquisition strategy, support RFP preparations, review proposals, contribute to program development, and provide program oversight. The staff needs to maintain their skills and be familiar with the latest techniques over the course of the acquisition. A gap analysis should be performed periodically to identify any gaps and a corresponding training, rotation and development plans put in place to address the missing skills.

The extent to which the PMO has the appropriate level of skills it needs to execute the acquisition can be rated on a sliding scale that extends from one extreme to the other: Weak to Strong.

3.4.1.2 PMO Staff Capacity

In addition to having the appropriate skills as described in Section 3.4.1.1, the PMO also has to have the right number of people with those skills to plan and execute the acquisition. It is important to be able estimate how many people you need, with what skills, and at what time. For example, you may not need three or four test experts while you are preparing the acquisition strategy, but you may need that many to oversee the test phase. What skills does the program need when? How much control over staff capacity and the escalation process for resolving staffing issues that are out of your control does the program have?

The degree to which the staff members have worked together before must also be factored into the schedule. Are they already a team or do they have to go through the required teambuilding steps before they function optimally?

The extent to which the PMO has the staff capacity it needs to execute the acquisition can be rated on a sliding scale that extends from one extreme to the other: Inadequate to Adequate.

3.4.1.3 PMO Staff Stability

Permanent Change of Station (PCS) and Permanent Change of Assignment (PCA) are facts of life in the military. The program must take this into account when formulating its acquisition strategy. How does the acquisition strategy compensate for the fact that the average time an active duty person spends in a PMO is roughly 2-3 years? Or, that the higher the rank, the more likely a person will be reassigned? Programs also have to evaluate the stability of civilian employees or contractors supporting the PMO.

How will continuity and history be maintained for the program? What will prevent a new software architect from completely changing course? What will help a new integration and test lead do his or her job quickly and effectively without compromising the quality of the system? Planning for continuity in the event of personnel reassignment or general staff reduction is an essential part of acquisition planning.

The extent to which the PMO staff is stable can be rated on a sliding scale that extends from one extreme to the other: Low to High.

3.4.1.4 PMO Process Focus

A process focus is just as important to the acquisition organization as it is to the suppliers with which the acquisition organization works. Are there solid, repeatable processes or are things done in an ad hoc fashion? Is there a focus on improving their processes through the course of the acquisition? How do the PMO processes mesh with the supplier processes? The extent to which the PMO understands and uses process and process improvement initiatives during the acquisition can be rated on a sliding scale that extends from one extreme to the other: Weak to Strong.

3.4.2 Stakeholders Drivers

Stakeholders are individuals or groups with vested interest in the system. For example, end users, system integrators, acquirers, and sponsors are all stakeholders. Each of these stakeholders' concerns can add unique risks, many of which are discussed in other sections. For example, how willing are users able to adapt and learn a new system? How much experience does your integrator have with this type of system? How well staffed is the acquisition organization? How well do the sponsors communicate their requirements?

The stakeholders drivers described in this section include the number and diversity of stakeholders, the level of stakeholder engagement, and the level of stakeholder agreement.

3.4.2.1 Number and Diversity

The sheer number and diversity of stakeholders in the acquisition impacts the acquisition strategy: the more stakeholders the PMO has to interface with, negotiate with, and keep informed, the higher the risk for the acquisition. It is essential that the PMO have a good understanding of how many stakeholders are involved in the program. A complex set of stakeholders increases software risk for the following reasons.

There is increased probability of divergent expectations and requirements (see Section 3.5.1).

Tradeoffs will not have equal impact on all parties--there will be winners and losers.

Communication becomes more complex

Coordination takes more time; there will be multiple approval paths and associated bottlenecks.

The amount of education required to discuss software issues increases.

Acquirers must be careful to include all the relevant stakeholders, since omissions can affect drivers and risks. This does not imply, however, that all stakeholders should necessarily have an equal voice in each phase of the acquisition.

The number and diversity of stakeholders can be rated on a sliding scale that extends from one extreme to the other: Small to Large.

3.4.2.2 Level of Engagement

The level of engagement by stakeholders in the acquisition process can affect the acquisition strategy. One way to describe the level of engagement is by monitoring the scope of stakeholders' involvement, the quality of their input and feedback, and their responsiveness. In the worst case scenario, you would get lots of input in an untimely fashion that is of poor quality with a narrow view of the system. In the best case scenario, you would get lots of input in a timely fashion that is of high quality from a broad view of the system. How will the program ensure proper representation, opportunities for participation, and commitment to deadlines from stakeholders? The level of engagement can be rated on a sliding scale that extends from one extreme to the other: Low to High.

3.4.2.3 Level of Agreement

Stakeholder agreement is the amount of consensus among the stakeholders during the acquisition. Stakeholders should be proactively included in all phases of the acquisition to maximize buy-in and system acceptance. The impact of any change must be considered from all stakeholder perspectives. If you don't involve them, you won't gain commitment, which usually results in dissatisfaction with the system in some form, may cause delays in the program, and so on. Even if stakeholders are proactively included, there might be significant disagreement on the requirements for the system, the priority of the requirements, on the schedule, on the deployment plan, and so on. How does the acquisition strategy handle this?

Another aspect to consider is stakeholder turnover. Military stakeholders suffer from the same PCS and PCA factors as the PMO. Sometimes a change in a key stakeholder can force the PMO to revalidate decisions that were made earlier in the program.

As agreement decreases, risk increases for the program. The level of agreement can be rated on a sliding scale that extends from one extreme to the other: Low to High.

3.4.3 Supplier Capability Drivers

Acquisition planning does not stop after a contract award is made; it is a continuous process. Many programs fail to evaluate drivers that play an important role after the program is in execution. A category of drivers that is often not considered is the capability of the supplier over time.

The supplier is the team that is responsible for producing the system and its associated artifacts. Typically, in large government contracts this is usually one or more contractors, but that is not always the case. The supplier could be another government entity. In this section, we refer to suppliers as all parties responsible for supplying the system to the acquirers.

The drivers pertaining to supplier capabilities described in this section include supplier staff skills, capacity and stability, and performance to date.

3.4.3.1 Supplier Staff Skills

A supplier is awarded a contract based on its ability to achieve the objectives associated with a statement of work. For software-critical systems, expertise may be needed in all of the areas described in Section 3.4.1.1. Other expertise may also be needed, depending on the domain.

During program execution, an evaluation of the supplier skill set is critical so that the acquisition strategy can be adjusted to mitigate any potential risks in this area. For example, if the supplier lacks testing skills, consider hiring an independent test agency.

The extent to which the supplier has the skills needed to execute its responsibilities on the program can be rated on a sliding scale that extends from one extreme to the other: Weak to Strong.

3.4.3.2 Supplier Staff Capacity

In addition to having the right skills, the supplier also has to have the right number of people with those skills at the right time to meet its commitments on the program. The extent to which the supplier has the staff capacity it needs to meet its commitments can be rated on a sliding scale that extends from one extreme to the other: Inadequate to Adequate.

3.4.3.3 Supplier Staff Stability

Usually, the supplier does not have the same issues with PCSs and PCAs as the PMO. It does, however, have to manage the external pressure of the job market in order to retain key people. Software engineers, system engineers, and architects are usually highly sought after resources. The hotter the job market in certain locations, the more difficult it may be for the supplier to retain its personnel.

Other factors that cause attrition include promotions within the supplier organization, transferring to the latest "hot" program, poor program management, low morale, and significant overtime.

The extent to which the supplier's staff is stable can be rated on a sliding scale that extends from one extreme to the other: Low to High.

3.4.3.4 Supplier Performance to Date

As stated in Section 3.4.3.1, a supplier is awarded a contract based on its ability to achieve the objectives associated with a statement of work. Typically, some form of past performance criteria are used during the evaluation process. During contract execution, the PMO needs to ensure the performance of the supplier does not add additional risk to the program. As the program progresses, the acquisition strategy can be modified as performance weaknesses or strengths are identified.

The extent to which the performance of the supplier to date can be rated on a sliding scale that extends from one extreme to the other: Poor to Excellent.

3.5 Life Cycle Category

Life cycle-related program characteristics and associated strategy drivers quantify the extent to which characteristics of various life-cycle phases affect the software risk in the program. Life-cycle strategy drivers include product definition and specification, architecture and design, verification and test, deployment, operations and maintenance, and disposal.

3.5.1 Product Definition and Specification Drivers

A great deal of management attention is placed on the requirements setting phase because missing, vague or changing requirements tend to be a major cause of poor software development outcomes [GAO 04a].

The product definition and specification drivers described in this section include the volatility and understanding of the requirements, quality attribute definitions, and interoperability.

3.5.1.1 Requirements Volatility

Stable, fully defined, unambiguous, consistent, complete, and testable software requirements are rare. Having flexible requirements is not a bad thing--trying to fully define software requirements too early or trying to limit requirements changes in a changing environment may be riskier than allowing some level of flux. The acquisition strategy needs to accommodate the degree to which requirements can or should change and how that change is to be managed. For example, the acquisition could be broken into several phases to ensure you are ready to move forward and are ready to negotiate or compete the follow-on phase.

Requirements changes can come from multiple sources including new or evolving mission needs, new or evolving technology (refresh and obsolescence), modifications to budgets, new or evolving policies or mandates, and changes in the understanding of the users over time. The program needs to understand how volatile or how stable the requirements for the system (and software) are at any given time in the life cycle. The amount of requirements volatility has a significant impact on the acquisition strategy for the program. Programs seldom have requirements that are all stable or all unstable--some requirements are firm from the start, some cannot be defined until other things about the system are known, some requirements may be in a constant state of flux as technology, COTS products, and mission needs (or the understanding of what is needed) evolve. There may be problems identifying "all" of the stakeholders, or after they're identified, getting them involved with the tradeoff and decision process. What one stakeholder wants may contradict what another stakeholder believes to be the correct product. The implication is that the acquisition strategy may have to have characteristics of either extremes or compromise between them.

How volatile are the system requirements? How volatile are the software requirements? How volatile are the hardware requirements? If hardware requirements are stable but the software requirements are not stable this driver can be split into hardware requirements volatility and software requirements volatility since different acquisition strategies could potentially be applied to the hardware and software components of the acquisition. The same can be said for system-level requirements and hardware/software requirements. Over time, the volatility of the requirements should decrease and modifications to the acquisition strategy can be made. If the volatility of the requirements does not decrease as expected, this is a good indicator that scope is not well-defined and that there is requirements creep.

The extent to which a program's requirements are volatile can be rated on a sliding scale that extends from one extreme to the other: Low to High.

3.5.1.2 Requirements Understanding

As stated in Section 3.5.1.1, fully defined, unambiguous, consistent, complete, and testable software requirements are rare. Some common issues with the understanding of software requirements include

Requirements are specified at too high of a level and there is no mechanism to refine them. Vague requirements are subject to multiple interpretations.

Ineffective techniques are used to solicit the requirements from the appropriate stakeholders.

The operational concept is not sufficiently detailed. For example, the operational concept might only address a subset of the users; it might be too sparse, which leads to misinterpretation, or it might contain conflicting requirements.

Users try to extend the functionality of the system past the "need to have" state into the "want to have" state.

Interfaces are not well understood, different than documented, or not specified at all.

Stakeholders are not involved in the process or they are involved and the importance of their input is ignored or minimized.

Requirements are not measurable, with clear acceptance criteria.

A common vocabulary is not shared or there is not the same level of domain expertise between the users, acquisition team, and performance team/supplier. For example, the requirement "Display a negative number" might be interpreted by a developer as "use a minus sign" when in reality, the user is an accountant and is expecting parentheses to be used.

The extent to which a program's requirements are understood by the users, acquisition organization, and performance team/supplier can be rated on a sliding scale that extends from one extreme to the other: Low to High.

3.5.1.3 Quality Attribute Definition Quality

Quality attributes, or non-functional requirements, are vitally important in ensuring appropriate software architecture. Non-functional requirements are used to express some of the "attributes of the system" or "the system environment" [Leffingwell 03]. They are usually called the "-ilities," quality attributes, or general requirements. Examples include reliability, scalability, and maintainability. Not all quality attributes are "-ilities"; performance and security are two non-functional requirements or quality attributes. For example

How dependable does the system need to be?

What level of security does the software have to provide?

What availability is needed? How will this be balanced with needed maintenance time frames? Is one hour a month downtime for maintenance tolerable?

Since non-functional requirements are requirements, Sections 3.5.1.1 and 3.5.2.2 apply to this type of requirement. It is particularly common for non-functional requirements to be defined without measurable, clear acceptance criteria. For example, "the page loads quickly" is inadequate. How fast must the page load under what conditions on the system?

Too often, requirements are thoroughly defined for the system functionality but not for the quality attributes. Failure to define the attributes the system needs to fulfill puts the program at great risk. How do you know which quality attributes are important for your program? How will you verify that your program has addressed its quality attribute requirements adequately? The extent to which a program's quality attributes are defined and understood can be rated on a sliding scale that extends from one extreme to the other: Low to High.

3.5.1.4 Interoperability

Interoperability is the ability of a collection of communicating entities to (a) share specified information and (b) operate on that information according to an agreed operational semantics. It is a multi-dimensional aspect of system engineering. The scope is far greater than simply interoperability of data. It includes, among other things, the degree of coupling and ownership. It also includes interoperability at the organizational and management levels [Brownsword 04].

One model of interoperability examines three different interoperability dimensions:

Programmatic--Activities performed to manage the acquisition of a system. The focus is on the contracts, incentives, and practices such as risk management. It is very important to think about how your program will interface to other programs. Will there be any impact on your schedule due to delays on a program building a system/service you need?

Constructive--Activities performed to create and sustain a system. The focus is on architecture, standards, and COTS. How many interfaces are software-related and how difficult are they to implement? How will you govern and control the interfaces? How will testing be performed? Does the information mean the same thing in my system as in the other system(s)? Which is the best system service to use?

Operational--Activities performed to operate a system. The focus is on interactions with other systems (programs) and with users. How will the concept of operations be managed? How will consensus be reached on priorities? What is the appropriate mix of capabilities for your system? [Morris 04]

The number of interfaces and their characteristics can be rated on a sliding scale that extends from one extreme to the other: Simple to Complex.

3.5.2 Architecture and Design Drivers

The software architecture plays a key role in the overall system design because it

embodies the earliest software design decisions that have the greatest impact on the system and may be difficult to specify early in the program

largely determines a system's ability to achieve its desired qualities (i.e., non-functional requirements) such as performance, security, reliability, modifiability, and others

enables or inhibits systematic software reuse and COTS integration

provides a means to analyze and predict a system's behavior

enables the supplier to subdivide work appropriately

The software architecture must be developed in conjunction with the systems architecture and software issues must be included in systems engineering tradeoffs.

The architecture and design drivers described in this section include precedence, quality attribute constraints, technology readiness, legacy system considerations, and COTS/government off-the-shelf (GOTS)/reuse components. These drivers can be affected by other drivers. For example, policies and mandates described in Section 3.2.1 can constrain the architecture.

3.5.2.1 Precedence

The degree to which the system is precedented or unprecedented has a significant impact on the risk of a program. According to the Encyclopedia of Software Engineering, precedented systems are those for which

the requirements are consistent and well-understood

the system architecture (both hardware and software) is known to be adequate for the requirements

the acquisition and development teams have worked together to build a previous similar system [Marciniak 05]

Violation of one or more of these causes the system to be classified as unprecedented.

Clearly, we are better at producing some types of software than others. We can consistently produce small or common software applications. In fact, a commercial marketplace has developed to provide these software components. On the other hand, we are less capable of producing large, unprecedented systems where few ready-made components are available or where the complexity of creating a unified system from components is stretched past previous limits.

Major stumbling blocks for unprecedented software-critical systems include

ill-defined, incomplete, contradictory, and changing requirements

inadequate architecture

software and hardware architecture mismatches

lack of domain experience

Relatively simple systems present relatively little risk. Similar systems can be used as benchmarks for reasoning about the system underdevelopment. Unprecedented systems obviously present much greater risk in that there are no equivalent benchmarks from which to draw inferences. A program can only extrapolate from other related systems. The more unprecedented a system, the more risk for the program. The degree to which a system is precedented can be rated on a sliding scale that extends from one extreme to the other: Low to High.

3.5.2.2 Quality Attribute Constraints

The definition of quality attributes was discussed in Section 3.5.1.3. The system can have simplistic qualities and few consequences for failure, very demanding qualities and severe consequences for failure, or be somewhere in between. The quality attributes can put significant constraints on how the system can be architected and implemented. The extent to which a program's quality attribute requirements constrain the solution set can be rated on a sliding scale that extends from one extreme to the other: Low to High.

3.5.2.3 Technology Readiness

Developing software for a system depends on many technologies. Is the hardware mature enough for the software to be developed? Is the development environment including compilers, configuration management tools, and other development tools sufficient to support the software development effort? Developing software also depends on the specific operational context. For example, Java is fairly mature at this point, but there are still some applications where it is inappropriate to use.

The extent to which system technologies are mature enough for software to be developed can be rated on a sliding scale that extends from one extreme to the other: Immature to Mature.

3.5.2.4 Legacy Considerations

Legacy system considerations introduce many avenues for risk during an acquisition including how well the legacy system and its interfaces are documented and managed, the amount of change between a legacy system and its replacement system, backward compatibility requirements, and so on. These are just a few of the many considerations, but they highlight the risks that programs that have legacy requirements must address.

How well the legacy system is understood significantly influences risk. If a legacy system and its interfaces are poorly documented there could be many interoperability and replacement issues. Is the legacy system written in some outdated or obscure programming language? How accurate is the information provided? How hard will it be and how long will it take to obtain any additional information your program requires? Does the government have full data rights to the existing system? How long will you have to interoperate with a legacy system? How will you keep up with the changes to the legacy system?

There are also transition risks as you move from a working legacy system to a new system. Do the two systems need to interoperate for some time or does the new system completely replace the old system? Will there be a new user interface to learn? Are there any safety or security risks as a result? Are there any facilities issues caused by the transition needs?

The requirement to be backward compatible with a prior or legacy system adds its own set of risks, especially when the earlier system still exists and is undergoing changes. How will new requirements in the legacy system be coordinated and managed with the new one? Is an appropriate governance board in place? How different can the new system be from the prior one? How long will backward compatibility need to be maintained after deployment?

The complexity of legacy system aspects that must be considered can be rated on a sliding scale that extends from one extreme to the other: Low to High.

3.5.2.5 COTS, GOTS, and Software Reuse

COTS or GOTS product use within the system may be considered a low-risk alternative for implementing specific software driven portions of the product design. However, COTS/GOTS should be considered as risky as any other software. The off-the-shelf options should not be considered "low risk" simply because the product can be purchased or obtained as a pre-packaged unit. COTS/GOTS products are not always complete or completely tested and most often are not based on the requirements for the new system being built. Planning for COTS/GOTS use, including a maintenance strategy, is an important part of up-front program planning. Maximizing COTS/GOTS/reuse is often a goal in programs today. The implications of using COTS/GOTS/reuse must be fully considered. An acquisition program must have their "eyes wide open" to understand the benefits and mitigate the risks.

The use of COTS/GOTS can be beneficial but it does bring some challenges with it. Some key challenges include [Carney 03]

COTS products are driven by the marketplace, not the system context

built-in assumptions of end-user processes that may not match yours

licensing, data rights, and warranties

limited control of content or frequency of releases

limited visibility into product internals and behavior

varying architectural paradigms between the product and other system components

GOTS sustainment responsibilities and plans

The way reuse is defined, planned for (e.g., productivity savings, maintenance), and managed determines the risk involved with reuse. Effective reuse typically involves more than just source code; design, requirements, test scripts can all be reused as well. The degree to which the characteristics of the program's intended operational context are similar to the operational context for which the original code was intended is one determinant of the risk involved with reuse.

Code reuse can come in many forms including reuse of legacy code and reuse of code between components/modules of the system being acquired. One must also take into account that such reuse is rarely ever "pure" (meaning that the code is reused as-is without any modification). More often the reused code is modified in some way to meet the specific requirements of the system under development or the reused must be "wrapped" by newly developed code to provide a consistent interface to the rest of the system. In either case, there is new development involved with the reuse and the amount of that new development will offset (at least partially) the benefits of the reuse.

The extent to which COTS/GOTS/reuse is planned to provide system functionality and realized over time can be rated on a sliding scale that extends from one extreme to the other: Low to High.

3.5.3 Verification and Test Drivers

The verification and test drivers described in this section include the complexity and availability of the test environment and the number of system configurations.

3.5.3.1 Test Environment Complexity

Typically, as the test environment complexity increases, the risk also increases. Do you need a special environment to perform software testing such as an aircraft? Is a classified test environment required? Is destructive testing required? Is the software part of a self-destruct capability of an end item? How complex are the test tools? Has appropriate training been provided? Is the test environment spread over several geographical locations that must be connected via a network? Are some of the elements of the test environment owned by other entities, for example, a government laboratory? Will simulations (of nascent hardware or of battlefield conditions) need to be developed?

The complexity of the test environment can be rated on a sliding scale that extends from one extreme to the other: Low to High.

3.5.3.2 Test Environment Availability

Typically, as the availability of the necessary test environment decreases, the risk increases. Can you get the testing time required in the necessary test environment? What are the priorities and schedule constraints on the test agency and facilities? How long does it take to get into the test cycle? Does this match up with your schedule? Is another program ahead of you using the test environment? What is the likelihood of slippage into your schedule time?

In addition to the agency and facilities for testing, you also have to consider the components with which you need to test. For example, if you performed some testing that interfaces with an external element that is a prototype or engineering model, what is the risk that the final version of the element will differ from what you tested with?

The availability of the necessary test environment can be rated on a sliding scale that extends from one extreme to the other: Low to High.

3.5.3.3 Number of System Configurations

The number of system configurations has an impact on the test requirements and how they will be tested. Tradeoffs will need to be made regarding the test requirements of the different configurations given the amount of funding available for testing. Can the same test cases be used or do different test cases need to be developed? What about test simulators? Can they be parameterized or do they need to be different?

Even if you only have one configuration under test, ensuring that the configuration is not modified during the testing process is not always easy. For example, a testbed was used during the day for testing the system configuration on a project. In the evening it was used for training purposes. Between the daily testing cycles, the test system configuration was changed by the training team without the test team's knowledge. How valid were the tests? The importance of managing the configuration of the system, test environment, and other test artifacts is essential.

The number of system configurations that need to be tested can be rated on a sliding scale that extends from one extreme to the other: Few to Many.

3.5.4 Deployment Drivers

The deployment drivers described in this section include the number of sites and system configurations, user and maintainer readiness, and data migration considerations.

3.5.4.1 Number of Sites

Sites may have different environments (hardware, software, networking, etc.) that impact the deployment. Personnel may have different levels of expertise at different sites. The approval process or acceptance process may be different. Another factor to consider is the timing for the deployments if there are multiple sites. If there is schedule overlap, staffing considerations must be taken into account.

The number of sites the system will be deployed to can be rated on a sliding scale that extends from one extreme to the other: Few to Many.

3.5.4.2 User Readiness

How ready the users are to accept and use the system is critical to the success of the system in fulfilling the mission. Some software considerations include

Have all operational software training needs been identified?

Will any new training facilities or simulators be needed?

Does the life cycle support the development of software for training assets?

Are the organizations impacted involved throughout the life cycle?

Will there be changes in personnel allocations to operate/maintain the system?

The extent to which users have the skills and desire to use the system can be rated on a sliding scale that extends from one extreme to the other: Low to High.

3.5.4.3 Maintainer Readiness

How ready the maintainers are to operate and maintain the system is also critical to the mission. Some software considerations include

Have all maintenance software training needs identified?

Will any new tools be needed? Does the life cycle support the development of software for the new tools?

Does the life cycle support the development of software for training assets?

Are the organizations impacted involved throughout the life cycle?

The extent to which maintainers have the skills, tools, and desire to operate and maintain the system can be rated on a sliding scale that extends from one extreme to the other: