Software Product Line Vision for Defense Acquisition, A
Grady H. Campbell, Jr. CMU/SEI-2002-TN-002June 2002
Product Line Practice Initiative
Unlimited distribution subject to the copyright.
Acknowledgements
This note is the result of many discussions, comments, and suggestions among members of the Business and Acquisition Guidelines group of the Product Line Practice Initiative at the Software Engineering Institute. In particular, John Bergey, Matt Fisher, Albert Soule, Larry Jones, and Sholom Cohen provided many insights on the complexities of the Department of Defense's acquisition policy and practices and many helpful comments on each draft. Joe Batman performed a detailed independent review on the second draft that contributed to smoothing and clarifying many points. Reviews by John Goodenough, Jack Ferguson, and Brian Gallagher provided key insights on several difficult issues. Linda Northrop wrote Appendix A, which describes SEI product line activities.
1 Introduction
Acquisition is the means by which the Department of Defense (DoD) obtains products and services that enable DoD mission capabilities. DoD Directive 5000.l [DoD 00] defines policies and principles that govern the operation of acquisition programs under the Defense Acquisition System. It identifies five major categories of policies and principles and calls for improvements in practices that contribute to them:
- achieving interoperability
- rapid and effective transition from science and technology to products
- rapid and effective transition from acquisition to deployment and fielding
- integrated and effective operational support
- effective management
As argued by Bergey, Fisher, and Jones [Bergey 99], a software product line approach would support many of the goals of DoD Directive 5000.1,1 and nothing in DoD policy precludes the adoption of such an approach to acquisition for software-intensive systems. However, nowhere in DoD policy is there any mention of a product line approach as a legitimate option, nor is there any guidance on when or how to undertake such an approach.
The goal of this note is to extend Bergey, Fisher, and Jones' position by suggesting how current policies and practices can be minimally changed to promote the appropriate consideration by acquisition programs of a software product line approach. These changes will lead DoD policy to overtly acknowledge the option for a product line orientation, eliminating doubts that program managers may have regarding the legitimacy or legality of a product line approach to acquisition. In addition, these changes will establish what acquisition programs must do to justify and institute a product line approach for acquiring software-intensive systems. The near-term result will be to make the adoption of a product line approach more viable for DoD acquisition managers who already see its applicability and potential benefit to their programs. The long-term result will be the diffusion of a product line perspective that will help acquisition programs to better accommodate diversity and change in DoD mission needs that they support, leading to faster deployment of more effective systems.
This note briefly describes the basics of DoD acquisition in Section 2 and argues in Section 3 for consideration of a product line approach to acquisition. Section 4 describes a near-term vision for acquisition based on a product line perspective. Section 5 details how that vision can be realized, by describing four narrow changes to DoD acquisition policy and practices. Section 6 outlines the elements of an enabling transition strategy. Appendices provide additional background information supporting a product line approach to acquisition: Appendix A describes related work by the Software Engineering Institute (SEI), Appendix B characterizes the life-cycle activities of an acquisition program from a product line perspective, and Appendix C outlines a long-term vision for acquisition based on product lines.
2 Basics of DoD Acquisition
An acquisition program is an effort to institute a new or improved operational capability in response to identified mission or business needs [DoD 00]. An acquisition program deploys products and services into an operational environment to enable or support the effective performance of a corresponding mission. The driving concern of DoD acquisition is to ensure a fair and effective competitive selection of a government or industry supplier that will provide the best value toward addressing targeted needs.
An acquisition program is initiated when an organization identifies a need¾a perceived deficiency or technological opportunity related to the organization's ability to perform its mission. An organization describes its needs in a Mission Needs Statement (MNS). DoD Instruction 5000.2 [DoD 01] prescribes how the defense acquisition system operates in accordance with DoD Directive 5000.1. It specifies an acquisition management framework comprising four phases:
- Concept and Technology Development¾Based on an MNS, identify and evaluate alternative solution approaches; identify and initiate prerequisite technology developments; and specify baseline requirements for the needed mission capability.
- System Development and Demonstration¾Develop and demonstrate a system for the production and deployment of products that will enable the specified mission capability.
- Production and Deployment¾Use provided system development capabilities to build products that support specified mission needs and transfer these products into operational use.
- Operations and Support¾Provide support for the sustainment and effective use of deployed products.
DoD acquisition policy defines the rules for acquiring products that span the gamut of size and complexity. Although this policy allows for streamlining when acquiring smaller or simpler products, its primary concern is large, complex, possibly replicated products that will remain in use for many years. Even though such products may have to allow for tailoring to different uses and most must be modified as needs change, DoD policy offers little guidance on how these concerns affect an acquisition. The nature of most acquisitions is a multiyear commitment to a selected supplier and a particular solution approach to the identified mission need.
Acquisition policy as traditionally conceived is based on a key assumption that reduces its effectiveness: the conception of a system as a fixed-point solution to a well-defined problem. This is starting to change with the introduction of evolutionary acquisition, which recognizes that problems are often poorly understood and incompletely described, and that solutions must be recreated and redeployed iteratively, as understanding improves. In addition, even as development proceeds, actual needs can continue to change, justifying the revision of products upon or even before deployment.
A product line approach to the acquisition of software-intensive systems goes beyond both traditional and evolutionary approaches to recognize that a single statement of mission needs may in fact represent a need for multiple solutions, each of which should be derivable in customized form from common assets. Such solutions can be developed more efficiently as a set rather than individually. Appendix A provides further information about work at the SEI toward establishing the principles and practices of a product line approach. The following discussion explores the prospect for a vision of software-intensive systems acquisition based on product lines.
3 Justifications for a Product Line Alternative
Many DoD missions depend on systems for which requirements and enabling technology change over time and which are deployed at multiple sites and in multiple versions. A separate or modified solution for every circumstance can be wasteful and time-consuming. Instead, such missions could be better served if acquisition programs were to take a product line perspective to explore whether and how different needs, operational contexts, solution technologies, and potential changes could be anticipated and addressed through an ability to develop and deploy different solutions. By exploiting the similarity inherent in alternative solutions to perceived mission needs, a product line acquisition could create a capability for the rapid production, deployment, and evolution of multiple products, each customized to suit specific needs.
Changes in acquisition policy and practices to accommodate a product line approach need not conflict with existing policy and practices. Accommodating a product line approach requires only a few specific additions and elaborations of current guidance. Acquisition reform trends represented by DoD Directive 5000.1 are beneficial to and mutually compatible with these changes.
As Bergey, Fisher, and Jones noted, a product line orientation is an effective response to several concerns raised in current acquisition policy. DoD Instruction 5000.2 prescribes the following:
- "Keep all reasonable options open and facilitate cost, schedule, and performance trades throughout the acquisition process." (paragraph 4.7.2.1.1.1)
- "Avoid early commitments to system-specific solutions, including those that inhibit future insertion of new technology and commercial or non-developmental items." (paragraph 4.7.2.1.1.2)
- "Define requirements in broad operational capability terms." (paragraph 4.7.2.1.1.3)
- "Evaluate how the desired performance requirements could reasonably be modified to facilitate the potential use of commercial or non-developmental items and components." (paragraph 4.7.2.1.1.5)
- "Programs with software components must be capable of responding to emerging requirements that will require software modification or periodic enhancements after a system is deployed." (paragraph 4.7.4.1.2)
These concerns are typical motivations for adopting a product line orientation. In addition, they suggest the possibility of taking a product line perspective even in a point-solution situation and maintaining it until changing, unsettled, or still-emerging requirements and technology can be resolved. A product line perspective aids in exposing, organizing, and resolving key requirements and design choices, leading either to the targeting for development of a preferred point solution or to the realization that mission needs actually warrant a product line approach.
4 A Near-Term Vision for Acquisition
The near-term vision for software product lines in
acquisition is that the DoD will officially support
program managers who decide to adopt a product line
perspective. The five goals listed in Table 1 represent enhancements to DoD
acquisition policy that are designed to foster a DoD
environment that would be receptive to product line
acquisitions.
Table 1: Near-Term Goals for Product Lines in Acquisition
| Goal | Enhancement | Rationale |
| NV.1 | DoD acquisition policy, procedures, and training provide appropriate guidance to acquisition practitioners on when and how to institute a product line approach. | Acquisition managers rightly hesitate to commit to a product line approach without supportive guidance. Such guidance will also help to ensure the consistent application of sound decision criteria and practices. |
| NV.2 | Acquisition programs give due consideration, during the Concept and Technology Development phase, to the suitability of a product line approach for their mission needs. | Unless program managers look specifically at the potential for applying a product line approach, most will never notice the opportunity. An analysis of product line approach viability will give any program team a better understanding of variability and change as aspects of targeted mission needs. A secondary reason for this goal is that such analyses will increase awareness of the product line concept among acquisition practitioners. |
| NV.3 | The DoD works cooperatively with the defense and software industries to characterize its future software needs in product line terms. Acquisition programs focus their research and development (R&D) funds on future mission product line needs to ensure the greatest benefit. | The DoD will benefit if it can influence industry to invest toward meeting future needs. Industry will risk investing in capabilities to meet future DoD needs if the DoD is able to characterize anticipated future directions. Without the benefit of such shared insight, forward-looking investments are difficult to justify. An acquisition program's R&D funds are one means for focusing industry attention on specific areas of future need. |
| NV.4 | The implications of a supplier having an existing product line capability are given proper weight in the selection of a source for satisfying mission needs. | A supplier who has made a business commitment to building the type of products needed by an acquisition should be preferred over one whose commitment is limited to contractual obligations. A business commitment is evidenced not just by past products developed on contract but also by discretionary investments and the retention and application of expertise across contracts. A supplier who has developed and can demonstrate an applicable product line capability is likely to offer better value to an acquirer. In addition, if selection criteria properly weights domain-focused investment and expertise, industry will have an incentive to allocate key resources of a business area to the exploration and development of DoD-directed product line capabilities. This will increase the range and quality of the offerings that are available to future acquisition efforts. |
| NV.5 | DoD legal and financial guidelines allow for the proper treatment of supplier product line investments and derived product pricing. | Investments by suppliers toward creating product line capabilities that are beneficial to the DoD are encouraged if such costs are ultimately recoverable through subsequent sales of derived products. If legal and financial practices inhibit investment or preclude recovery, suppliers will resist creating capabilities that would reduce their ability to maximize direct costs. On the other hand, with cost-recovery rights, competitive-pricing pressures will encourage vendors to make product line investments as a cost of doing business. This has the potential to improve the cost, quality, and timeliness of acquired products without unacceptably compromising profits for suppliers. |
Proposed policy enhancements for achieving the goals listed in Table 1 depend on a key choice faced by each product line acquisition program: whether the form of the acquisition will be direct or indirect.
- In a direct product line acquisition, a government-owned product line capability is developed and used to create needed products.
- In an indirect product line acquisition, needed products are obtained using a supplier-owned product line capability.
The policy extensions proposed here anticipate both of these forms. The key difference between the two is whether the acquirer or the supplier incurs the primary financial risk and potential for reward. Current policy (starting with paragraphs 4.2.3, 4.3.3, and 4.4.3 of DoD Directive 5000.1) provides guidance on how an acquirer would choose between these two forms to ensure best value over the life cycle of a targeted mission capability. The two forms are equivalent in their potential for improved per-product cost, quality, and responsiveness to changing needs over a mission-capability life cycle. They differ in how and why the needed capability for building products comes into being, with consequent implications for the control of scoping, capital-investment costs, ownership rights, and flexibility for subsequent evolution.
Having established the utility of a product line acquisition, an acquirer would choose the direct form when suppliers cannot satisfy identified needs-that is, when
- No supplier has an existing product line capability that allows needed products to be produced at an acceptable cost and schedule.
- No supplier is willing to finance the development of a suitable product line capability in return for anticipated future product acquisitions.
By directly acquiring a product line capability instead of a corresponding set of point-solution products, the acquirer can leverage funds and expertise across the entire product line to reduce maintenance costs and increase flexibility to address unanticipated needs. Based on normal best-value considerations, the elements of the envisioned product line capability may be developed either directly by the government or through contracts with industry suppliers. Either way, the resultant cost and schedule for producing individual products will be reduced to the marginal cost of using the product line capability, which should be targeted at one to two orders of magnitude less than that required for an equivalent point-solution product.
With a direct product line acquisition, the acquirer undertakes a significant development-phase, life-cycle investment, potentially on par with the cost of developing two or more point-solution products. In contrast, in an indirect acquisition, an industry supplier accepts responsibility for investments in product line capabilities. In return for this investment, the supplier gains control of any resulting production cost savings. As a consequence, an indirect acquisition tends to reduce the acquirer's pre-deployment risk and cost relative to a direct acquisition. This is because, from the acquirer's perspective, the indirect form is superficially indistinguishable from a point-solution approach in that the scope of the acquisition can revert to contracts for the delivery of individual products. Nevertheless, the supplier's investment in developing a responsive product line capability makes the needed products obtainable at a reduced cost and schedule. The supplier can choose to translate per-product cost savings into either an increased return on investment (profit) or reduced prices, depending on the competitive situation and the degree to which DoD funds contributed to the capability's development.
When an industry-developed product line capability is insufficient for building products exactly as needed, an indirect acquisition is somewhat more complex. This problem is resolved either by rethinking needs in order to target producible products that will suffice, or by asking for enhancements to the existing product line capability. In the case of enhancements, the acquirer may either provide development funding or choose to incur higher per-product costs in consideration for the supplier's funding of enhancements. Program managers should consider the implications of ownership rights and responsibilities in deciding whether a product line capability and enhancements should be funded by the acquirer or by a supplier.
5 Perspectives for Policy Enhancement
Achieving the near-term vision requires enhancing aspects of existing acquisition policy and the corresponding acquirer guidance and training. Needed policy enhancements will address questions relating to four perspectives of acquisition and questions associated with each:
- acquisition-program life cycle. How can the life cycle of an acquisition program be characterized so that it accommodates both point-solution and product line approaches to acquisition; leads acquisition efforts to choose the better approach for their needs; and applies equally well to both software-intensive-system and software-only acquisitions?
- source selections. How can suppliers' product line qualifications, including relevant specialized expertise and past investment in applicable product line capabilities, be properly evaluated to ensure that the best-value alternative is chosen?
- industrial base. How can a focus by acquirers on product lines lead to a more responsive industrial capability that can rapidly provide lower cost products that anticipate emerging DoD mission needs?
- funding models. How can DoD funding models be applied to match the economic profile of product line acquisitions that depend on
- investment in production assets and infrastructure for saving production time and cost?
- reimbursement of direct effort for adding and refining capabilities to address unanticipated mission-specific needs?
Below, the effects on acquisition policy are discussed further from each of these perspectives. The described revisions alone are sufficient to promote the institution of product lines as a viable option for future acquisition programs. These changes are conceived as extensions to existing policy that will coexist with continuing improvements related to point-solution acquisitions. Specifically, policy extensions anticipate that a product line approach to acquisition will
- conform to the acquisition life cycle, milestones, and reviews defined in DoD Instruction 5000.2
- accommodate either single-step or evolutionary product-deployment strategies
- be applied within the scope of existing acquisition programs without requiring a closer integration of programs targeting similar mission needs
- maintain the existing relationship between the DoD and industry with proper adherence to rules for source selections and the evaluation of supplier performance
- accommodate sound risk-management practices for detecting and addressing acquisition and development problems
- support an emphasis on experience-based continuous improvement of acquisition and development practices
The nature of a product line acquisition is discussed below from each of the four perspectives on acquisition policy. To elaborate on each perspective, a "DoD policy prescription" characterizes corresponding proposed revisions and enhancements to DoD acquisition policy and practices.
5.1 Acquisition-Program Life-Cycle PerspectiveThe acquisition management framework as defined in DoD Instruction 5000.2 is a suitable guide for both point-solution and product line acquisitions. As shown in Figure 1, superimposing the phases of this framework onto the general form of a product line approach provides a reasonable model for the life cycle of a product line acquisition program. Appendix B discusses this life cycle and its activities in more detail and describes how each phase of the acquisition management framework can be interpreted from the perspectives of both point-solution and product line approaches for comparison.
Figure 1: An Acquisition-Program Life Cycle
The life cycle proposed here includes an activity for each of the phases defined in DoD Instruction 5000.2, with the addition of two small but significant refinements:
- The outcome of the Production and Deployment phase may be a set of similar products, each customized to different needs, rather than just one product.
- Dashed lines show the potential for feedback among phases leading to changes or enhancements of earlier results.
The first of these refinements comes with choosing to perform a product line, rather than a point-solution, acquisition. A point-solution acquisition produces a single product, either in a single step or in increments (in the case of an evolutionary acquisition). In contrast, a product line acquisition produces a set of distinct but similar products, each customized and deployed to meet different needs. (Appendix B describes other significant differences within each of the life-cycle phases between a point-solution and a product line approach. For example, a product line approach results in a set of products because, in contrast with a point-solution approach, it prescribes the development of a production capability, which includes a set of core assets that can be reused to create different products.)
The second refinement made in characterizing the proposed acquisition-program life cycle is the potential for phases to be repeated. In single-step acquisitions, the phases of the life cycle are performed once. In evolutionary acquisitions, the phases may be performed repeatedly in the prescribed order for the incremental delivery of a final product. In product line acquisitions, any phase may be repeated multiple times based on feedback from a later phase, or to accommodate needs that are changing over time or that are associated with different products.
DoD Policy Prescription for the Acquisition-Program Life-Cycle PerspectiveExtend DoD acquisition policy to provide guidance on when and how to adopt a product line approach to acquisition. This guidance needs to better accommodate both point-solution and product line approaches to acquisition, reduce perceived differences between system and software-only acquisitions, and guide acquisition efforts in choosing the best approach.
- Every acquisition program will evaluate, during Concept and Technology Development, whether a product line approach would fit the needs stated in its MNS. Acquisitions that warrant the production and deployment of multiple customized products would choose a product line approach. Qualifying programs should perform a cost-benefit analysis of the tradeoffs in choosing a product line over a point-solution approach.
- In developing its acquisition strategy, each acquisition program that adopts a product line orientation will determine whether it should initiate an indirect acquisition (industry can provide a suitable initial product line capability) or a direct acquisition (the acquisition must develop that capability). An indirect acquisition will require System Development and Demonstration phase activities only if the initial capability requires enhancements to meet specific mission needs. A direct acquisition requires comprehensive System Development and Demonstration phase and Production and Deployment phase activities.
- In a product line acquisition, unlike most point-solution acquisitions, software requires distinct Development and Demonstration phase and Production and Deployment phase activities. System Development and Demonstration phase activities are performed to create the assets and mechanisms that can be used to produce any product within the scope of identified mission needs. Production and Deployment phase activities are performed, repeatedly applying those assets and mechanisms, to deliver multiple specific products or product versions into use as needed.
- During Concept and Technology Development, a product line acquisition program will define its requirements in a form that indicates differences (variabilities) among the envisioned set of needed products. This provides the means for identifying varied system-usage contexts and anticipated future mission and technology changes. Assumptions regarding variabilities indicate conditions and choices that may warrant different solutions to a problem. These assumptions then establish the basis for creating a capability for building alternative solutions to meet differing and changing needs. Programs for which all variabilities can be eliminated will revert to a point-solution approach.
In an acquisition of a point-solution system, acquirers seek a supplier with the knowledge and expertise needed to build an appropriate solution, perhaps requiring months or years of diligent preparation and effort. Offerers may not necessarily have built similar products before or initially know the possible solutions or tradeoffs to consider for the particular problem being presented. A supplier is selected based on well-founded but ultimately subjective evaluations that the acquirer makes about the offerers' claims concerning
- past performance on other efforts, preferably in solving similar problems
- the nature and maturity of their development process and associated management and engineering practices
- how they would approach developing a solution to the current problem
- their estimates of the time and cost required to obtain a solution
- known uncertainties and risks that could impede a solution and the offerers' proposed mitigation strategies
In a product line acquisition, a basis may exist for increasing the objectivity of offerer evaluations. When offerers have previously built similar point-solution products or have an existing product line capability, they may have the ability to rapidly build low-cost prototypes that approximate needed products. Subjectively qualified offerers may be asked to demonstrate their expertise and ability to build responsive products by producing any one or more of a prescribed set of representative prototypes.
Competitive prototypes as an element of a source selection are beneficial in that they
- increase the objectivity of the acquisition by giving a concrete basis for evaluating competing offers
- validate that offerers have the knowledge, expertise, and productive capability to produce the type of products needed
- increase the likelihood of choosing an offerer who has the best initial understanding and productive capability and is therefore closest to creating a fully compliant solution with the least risk and cost
- reveal strengths and weaknesses among offerers that may justify more narrow selections for specific sources of expertise in developing particular component capabilities
- provide a concrete basis for comparing and resolving risks concerning alternative solution approaches before committing to a particular approach
- increase confidence in proposed solutions and time/cost estimates and focus attention on less understood solution elements
- establish concrete interpretations of requirements specifications enabling clarification and refinement
Source selection for the System Development and Demonstration phase of a product line acquisition will differ depending on whether the acquisition strategy is direct or indirect. For direct product line acquisitions, the purpose in this phase is to obtain assets and mechanisms that comprise the needed production capability. Offerers are evaluated based on specific, relatively narrow expertise that is relevant to particular elements of production capability being sought. In the case of indirect product line acquisitions, the purpose in this phase is to gain access to an existing capability and extend it as needed to support government needs. Offerers are evaluated based on the degree to which their existing capability matches government needs and the projected costs to extend it.
DoD Policy Prescription for the Source Selections PerspectiveExpand the guidance for source selections, with a particular focus on software, to promote the use of demonstration prototypes for more objective evaluations of competing suppliers' product line capabilities. Prototypes provide a concrete basis for evaluating and comparing the capabilities of alternative suppliers based on objective criteria. Furthermore, a requirement that each offerer be able to demonstrate an ability to produce any of a specified set of prototypes can provide the basis for determining whether the offerer has an appropriate product line capability (knowledge, expertise, and ability to perform). A requirement that each offerer be able to produce multiple differing prototypes would reward prior product line investments, increasing the incentive for industry to make such investments in the future.
- Acquisitions having a product line orientation will evaluate the value and utility of requiring qualified suppliers, based on initial proposals, to produce an operational prototype that demonstrates their existing production capability, their understanding of targeted mission needs, and the viability of their solution approach.
- In qualifying offerers for prototype-based competitions, acquirers will consider both evidence of past performance in building similar solutions and prior investment in product line capabilities targeted toward building such solutions. Such evaluations will give substantial weight to a demonstrable ability to build any of a range of alternative prototype solutions.
- Acquisitions requiring prototype-based demonstrations will define a specific scope and evaluation criteria to be satisfied in the competition and precise time/expense ceilings. The goal will be to evaluate and compare the levels of mission capability that each supplier is able to achieve within a fixed time and cost budget. Evaluators will consider each offerer's ability to deliver multiple alternative solutions corresponding to a limited range of differing problems. The acquirer may need to provide limited funding for demonstration efforts, but should do so under very strict time and cost limits consistent with satisfying specific evaluation criteria.
- Criteria for evaluating suppliers will be weighted to reward prior or concurrent investment by a supplier in an applicable product line capability. This weighting should consider the nature and extent of an offerer's demonstrated ability to create different solutions in response to prescribed variabilities in targeted needs.
- Criteria for evaluating suppliers having existing product line capabilities will include the scope of the capabilities relative to specified product line requirements; the time efficiency of using those capabilities to build a specified set of demonstration solutions; and the correctness of the solutions relative to specified verification and validation criteria.
The DoD depends on industry for much of the expertise and productive capability necessary to build systems that enable the timely and effective performance of organizational missions. In turn, the health of organizations comprising the defense industrial base depends in many cases on their ability to win DoD business directed toward building those systems.
This mutual dependency means that investments in infrastructure benefit both the DoD and industry. The need for such investments in hardware-manufacturing facilities, before products are needed, is accepted. Analogous investment in targeted software-production capabilities has comparable benefits in reducing costs and increasing the efficiency and effectiveness of development resources. Such investment requires DoD and industry to refocus on software-expertise specialization as a key factor in successful development efforts. Not only do commercial software, information systems, and embedded systems require different types of software expertise to build, but different types of embedded or information systems require additional specialized knowledge and experience. To succeed, development efforts require either a core of people who have previously built similar systems or great expense to create such knowledge and experience through trial and error.
The existing perception of the DoD software industrial base is amorphous and fails to distinguish the various types of expertise-based competencies that may reside in different organizations. Most of the focus on expertise relates to the technologies used rather than the mission capabilities supported. The DoD software industry needs to be reconceived to categorize potential suppliers relative to the capabilities needed to perform future missions. This focus on mission-capability expertise can then provide acquirers with the basis for targeting and funding the development of product line capabilities that will support future mission priorities. This will then create the foundation for reducing the cost and increasing the timeliness and effectiveness of future acquisitions in targeted mission areas.
The complexity of DoD acquisition policy is a reflection of the size and complexity of many of the products to be acquired and the challenges of producing them. Human and technical resources must be developed before an acquisition's targeted capability is needed. The expertise for creating such products must be nurtured and sustained during times of low demand. The uncertain and highly variable funding of current products or the reliance on commercial demand to finance the industrial infrastructure needed for future products does not properly serve the DoD's needs. The costs of creating and maintaining capabilities that will support future needs must be met by well-considered investments that do not lead directly to products, but instead enable efforts that are needed to streamline the building of future products.
Investment in the DoD software industrial base can occur prior to or as part of an acquisition. Prior to and during the initial phase of an acquisition, the DoD provides funds for needed technology R&D. Within a product line orientation, this funding would focus on developing capabilities that are specifically supportive of anticipated future mission needs and emerging technological opportunities.
Within an acquisition that involves extending a supplier's existing product line capability, government funds can be allocated to perform needed enhancements. Funding such extensions can be justified as a legitimate investment in needed industrial capability in that it will provide a future ability to produce and modify needed products more quickly and at a reduced cost. Alternatively, a supplier may choose to fund extensions with the expectation that competition requires enhanced capabilities or that the extensions will lead to increased per-product profit margins, even with prices that are substantially lower than equivalent point-solution prices.
DoD Policy Prescription for the Industrial Base PerspectiveFocus technology investments on developing specialized software capabilities and expertise that will contribute to addressing future mission needs more effectively. Promote communication and cooperation across government and industry that contributes to advances in needed specialized capabilities.
- Each acquisition program, as part of its Technological Opportunities activities (paragraph 4.7.2.3 of DoD Instruction 5000.2), will identify and categorize the types of software capabilities required by its customers for the effective performance of their missions. Based on a survey and analysis of industry and government sources, a program can identify potential suppliers of specialized areas of software expertise and capability. Identified suppliers should be engaged in a continuing dialogue concerning potential future acquisition needs and the further development of their abilities to respond to those needs.
- Each program taking a product line orientation will direct funds for applied research and advanced technology to the development of product line infrastructures in software capability areas that are responsive to targeted future mission needs.
- Acquisition programs that adopt a product line orientation will designate future needs in terms of anticipated mission capabilities and variability assumptions (relative to customer needs and technology). During the Concept and Technology Development phase, a program may advise potential suppliers on directing business investments toward product line capabilities that will be responsive to future acquisitions. In the System Development and Demonstration phase, a program may fund selected suppliers to create responsive product line capabilities that reflect their assessments of potential improvements in the cost and timeliness of producing future products.
- Potentially applicable software capabilities that are available from commercial sources (specifically, dual-use technologies) will be analyzed for likely divergences from targeted mission needs. Such divergences must be evaluated as a risk that is attributable to inherent differences in commercial and military needs, and that may warrant the future development of mission-specific alternatives. A program may provide funding during the System Development and Demonstration phase to enable a supplier to adapt its capabilities to support government needs.
Federal acquisition regulations define two broad categories of contracts: fixed price and cost reimbursement. Either type will suffice for point-solution and direct product line acquisitions. For both types, selected suppliers are contracted to deliver products having specified capabilities.
For indirect product line acquisitions, cost-reimbursement contracts will not be suitable from a supplier's perspective. A substantial portion of a supplied product's value may derive from the product line capability, with relatively low levels of effort required to transfer that capability into a product. A cost-reimbursement contract does not provide a means for suppliers to recover their prior investments in product line assets and mechanisms.
A fixed-price contract may suffice if a supplier has a product line capability that is highly responsive to identified mission needs. Alternatively, if needs are sufficiently flexible to permit accepting a best-fit solution, a fixed-price contract could suffice. These cases are analogous to acquiring a commercial off-the-shelf (COTS) product but offer additional flexibility for customization.
Fixed-price contracts will not suffice for the more common situation faced in indirect product line acquisitions, in which the best-qualified supplier has a product line capability that only partially matches identified mission needs. While that capability may be sufficient to quickly build prototypes or even early-increment-deployable products, it may not be sufficient to produce the complete range and variety of products needed over an entire mission life cycle. Such acquisitions require contracts that separately address both facets of these situations:
- What is the appropriate unit price for a product produced using a product line capability? Products that can be derived using preexisting product line capabilities may be obtainable at a predetermined price (fixed but dependent upon the features needed). Depending on competition, the unit price for products should be substantially less than the cost of building an equivalent point solution, but will usually be much greater than a conventional cost-reimbursement contract would justify (in that direct costs will be low and will not account for prior investments in the product line capability). This price may be set by market forces for dual-use-type products. Otherwise, the price must be negotiated based on estimates of the effort and costs saved by having access to a product line capability, and the imputed value to the customer of being able to obtain the resulting product rather than having to resort to less effective alternatives.
- How are enhancements to product line capabilities funded, and how does this option affect the unit price of products? If an acquisition depends on the supplier's willingness to make enhancements to a preexisting product line capability and the supplier funds those enhancements as a capital investment, the determined unit price may be adjusted to account for the increased value of products that exploit those enhancements. As an additional incentive, the supplier may be allowed to offer the enhanced capability to other customers without restriction. Alternatively, if the acquisition provides funds to make enhancements, the acquirer can either restrict other uses of the enhancements or negotiate a decrease in product unit prices as compensation. This choice may be influenced by the degree to which such funding is seen as contributing to a stronger industrial base. Typically, enhancements may be funded jointly, requiring that the resulting implications be negotiated.
- supplier funding. The supplier sees future market value in including these extensions in its product line capability and funds them as an investment of available capital. When products that require these extensions are acquired, their corresponding unit prices may be higher in light of this added investment and the products' greater value to their acquirers.
- acquirer funding. A supplier who lacks other customers for particular capabilities may not be willing to fund needed extensions. The acquirer can fund the supplier to add capabilities for the acquirer's exclusive use or may choose to fund limited after-delivery extensions to completed products. Even if the product line capability is extended to permit enhanced products, product unit prices should remain unchanged.
- shared funding. The acquirer and supplier are willing to share the cost and benefit of having an enhanced product line capability. The acquirer can benefit in being able to obtain enhanced products at a reduced cost. Alternatively, such funding may be justified as the least-cost means available to achieve the level of industrial capability needed to obtain mission-critical product capabilities. In either case, the supplier may benefit in being able to offer capability-enhanced products to other customers.
Establish funding-model guidance that accommodates the economic profile of indirect product line acquisitions. This will result in product pricing that reflects both the efficiencies of preexisting product line capabilities and the need to allow for mission-driven capability enhancements.
- Based on existing guidance for financing industrial-manufacturing capabilities for hardware, acquisition authorities will develop guidance for the funding of software-manufacturing capabilities during the System Development and Demonstration phase. They will also develop guidance for appropriate cost recovery by industry during the Production and Deployment phase as compensation for having privately funded such capabilities.
- Recognizing the potential benefits of cost sharing to the DoD, acquisition authorities will develop guidance concerning contracting practices when using a supplier's product line capability that lacks some capabilities needed to support mission needs. This guidance will prescribe how to properly coordinate the fixed-price acquisition of end products with cost-reimbursement contracts for required extensions to the supplier's existing production capability.
6 A Near-Term Transition Strategy
Current DoD policy on acquisition does not obstruct a product line approach. However, it does not explicitly promote or support this approach either. Nevertheless, the omissions and impediments identified here have been worked around within current policies by a few knowledgeable and risk-tolerant acquisition programs. The wider adoption of a product line approach to acquisition requires a greater awareness of the potential benefits and reassurance of the legitimacy and legality of such an approach by programs. Within the scope of the existing acquisition framework, both can be achieved through the modest extensions to DoD acquisition policy and practices suggested here. The full potential of product line
acquisitions for the DoD will require actions toward the long-term vision that is summarized in Appendix C.
Increasing the potential and reducing the risks of taking a product line approach to acquisition requires action relative to all four perspectives of acquisition. Prescribed actions focus primarily on policy revisions. Subsequently, these actions must translate into revisions in documented procedures and training that guide acquirers in their work. The focus of these initial revisions is only on increasing the perceived validity of a product line perspective for acquisition programs. The actual adoption of this perspective depends on the willingness of acquisition practitioners and industry to accept this perception. Such acceptance will take time but is inevitable if DoD acquisition policy, guidance, and training are revised to create awareness and eliminate unnecessary impediments.
In the overall scheme of DoD acquisition, promoting a product line orientation requires modest changes in DoD policy. While current policy does not preclude a product line orientation, by not explicitly identifying the possibility, acquisition managers are left with doubts about its acceptability and whether and how to proceed properly. The transition required to achieve legitimacy for product lines has three themes:
- Influence and advise policy makers on including the appropriate language needed to establish that a product line orientation to acquisition is a legitimate option.
- Develop guidance for acquisition practitioners on any special considerations that can affect product line acquisitions.
- Advise and assist acquisition programs for which targeted mission needs may warrant a product line approach.
As experience with product line approaches to acquisition grows, the inherent potential of software/systems development as an engineering discipline will become more apparent. The inherent limitations of craft-based development will yield to the efficiencies and flexibility that are possible through an expertise-focused, mass-customization approach to the acquisition and development of software-intensive products.
Feedback and Contact
Comments or suggestions about this document or the series of technical notes on software product line business and acquisition guidelines are welcome. We want to meet the needs of DoD and government personnel who are involved in the business and acquisition aspects of implementing software product lines. To that end, comments concerning this technical note, the inclusion of other topics, or any other issues or concerns will be of great value in continuing this series. Send comments or suggestions to
Linda Northrop, Director
Product Line Systems Program
lmn@sei.cmu.edu
Software Engineering Institute
Carnegie Mellon University
Pittsburgh, PA 15213
Appendix A SEI Work on Product Lines
A software product line is a set of software-intensive systems that share a common, managed set of features satisfying the specific needs of a particular market segment or mission and that are developed from a common set of core assets in a prescribed way [Clements 02]. Organizations are finding that using a product line approach for software can yield remarkable quantitative improvements in productivity, time to market, product quality, and customer satisfaction. Software product lines allow these organizations to leverage their expertise and take economic advantage of the similarity among their products in a planned, predictable, and strategic way.
The SEI has produced a substantial body of work
related to software product lines. The SEI Web site for
the Product Line Practice Initiative
( Though the Framework provides a thorough description
of the management and technical practices necessary for
software product lines, many of those practices take on a
different light in a DoD acquisition setting. Another
Web-based document, Software Product Line Acquisition:
A Companion to A Framework for Software Product Line
Practice [Bergey 01] was
created to accompany the Framework and to specifically
address the government acquisition issues and needs as
they relate to the necessary product line practices.
The SEI has also developed methods that aid in specific
activities of a product line effort. The Attribute-Driven Design
(ADD) method was developed for defining product line architectures [Bachmann 00]. The SEI's Architecture
Tradeoff Analysis MethodSM (ATAMSM) [Clements
01] is frequently used to effectively evaluate product line
architectures. A product line analysis method [Chastek 01] that incorporates earlier
domain-analysis techniques with use-case analysis and requirements
engineering is available to assist in the early product line
activities. Options Analysis for Reengineering (OAR) [SEI] is an SEI method that can be used to evaluate
the feasibility and economy of mining existing components for a
product line.
In addition to these, the SEI has also written multiple technical
reports on specific product line topics and case studies,
collaborated directly with customers on product line efforts, held
more than a dozen workshops attended by experienced product line
practitioners from all over the world, and organized and sponsored
the First Software Product Line Conference in August 2000 [Donohoe 00].
Appendix B
A Life Cycle for DoD Acquisition Programs
The acquisition management framework as defined in DoD Instruction
5000.2 is an amenable basis for both point-solution and product line approaches. Figure 1 shows an equivalent formulation of a compatible acquisition-program life cycle, highlighting two insights:
The activities of a typical product line approach can be cast in a form that mirrors the phases of an acquisition-program life cycle:
DoD Instruction 5000.2 mandates the iterative
development of software (in paragraph 4.7.3.2.3.3.2). It
also prescribes evolutionary acquisition as a preferred
strategy over the traditional single-step approach (in
paragraph 4.7.3.2.3.3.1). Iteration among life-cycle
phases accommodates incompletely understood or changing
requirements. Iteration has even greater significance
within a product line acquisition in which multiple
products and versions are to be supplied, potentially in
response to elements of mission need or technology that
may not be known initially or that may change at some time
during the life of the acquisition program.
This means that phases are managed to permit the iterative transition of each phase's baselined results to subsequent phases, even as extension and refinement of that phase's results continue. An acquisition effort continues indefinitely, until its objectives are met, resulting in a product line capability that, by accounting for projected diverse and changing operational needs, is sufficient to build and maintain future products as needed.
The following descriptions characterize the four phases of the acquisition management framework as activities of an acquisition-program life cycle. It presents these from a software perspective and compares how point-solution and product line approaches differ for each. Discussion of each phase focuses first on the aspects that are common and then on those that are different between the approaches. The essential difference between the two approaches is the extent of diversity and the potential for change (implicitly or explicitly) anticipated in an MNS:
Note that a requirement for the post-deployment configurability of a product is an unrelated issue. The distinction between point-solution and product line approaches is entirely concerned with whether a need exists for multiple products or versions to be deployed, any of which could provide needed options for configuration at the time of deployment.
Concept and Technology Development is concerned with establishing requirements that correspond to specified mission needs of the end-user organization(s); evaluating opportunities for exploiting existing and emerging technology; and identifying and evaluating alternative solution concepts. Evaluating alternative solution concepts includes deciding whether a point-solution or product line approach responds best to the specified mission needs, establishing criteria for effective development practices, and defining an acquisition strategy that is appropriate for the selected approach. A product line approach is considered whenever mission needs warrant multiple products.
Point-Solution Approach: This approach is preferred when a single, statically defined product is sufficient to address identified mission needs. Identified needs are elaborated to create a preliminary specification of requirements. Alternative solution concepts are reduced and refined to define an appropriate system decomposition and identify required component technologies.
Product Line Approach: This approach is preferred when multiple similar products will be needed to address identified mission needs. Identified needs are refined into a specification for a range of requirements, representing anticipated solution diversity and change over time and corresponding to a family of products. The acquisition strategy addresses the definition and continuous improvement of product line development practices, including mechanisms for producing products customized to specific needs.
System Development and Demonstration is concerned with enacting an acquisition strategy to create and demonstrate a capability for the production of needed products. In the Production and Deployment phase of the acquisition program, this production capability enables the production and deployment of products that support specific needs.
Point-Solution Approach: The sources of parts that are needed in the production of a product are identified. Parts in small quantities, or a means of repeated construction for large-quantity replication, may be obtained or developed. One or more instances of the solution product are produced and demonstrated for correctness and completeness, relative to specified requirements.
Product Line Approach: Variabilities in requirements are formalized and referenced to create a design that specifies the range of needed software assets and how they are to be tailored and combined to construct alternative solutions. Specified assets and mechanisms will constitute a capability for the replicated production and deployment of customized solution products and associated materials and mechanisms. The means of obtaining needed assets, by acquisition or development, are established. Sample problems are posed for the creation and evaluation of sample products to demonstrate the viability of the resulting capability. Emerging changes in mission needs and technology guide the evolution of developed capabilities.
The nature of this phase of a product line approach differs further depending on whether the product line acquisition is direct or indirect:
Production and Deployment is concerned with delivering products that enable operational capabilities to address mission needs. Regardless of the approach taken, deployment entails installing and initializing each product in its operational environment, verifying its correct functioning, and training end users in its proper and effective use.
Point-Solution Approach: Production entails obtaining and composing parts to form a solution product and verifying whether the product satisfies specified requirements.
Product Line Approach: Production entails using the developed parts supply and production capability to produce products as needed, each tailored to specific customer needs. A product may include associated materials that support testing, installing, and using the product and training users. The requirements for each particular product are specified in terms of supported variability options. Each product must be verified relative to both common and variable aspects of those requirements. Discrepancies in needed production or deployment capabilities result in the reiteration of System Development and Demonstration phase activities for correcting or enhancing those capabilities.
Operations and Support entails inspecting, adjusting, and replacing consumable and degenerative parts, assisting end users with any problems they encounter in their use of a product, and identifying any inadequacies in light of actual end-user experiences.
Point-Solution Approach: Subsequent changes in the requirements or enabling technology result in the initiation of product modification and redeployment efforts as needed.
Product Line Approach: Subsequent changes in the requirements or enabling technology result in the reiteration of Production and Deployment phase activities to provide a replacement product.
Appendix C
A Long-Term Vision for Acquisition
Once the near-term product line vision for acquisition has
been accepted, it will be possible to move toward a more ambitious long-term vision. The long-term vision is that DoD acquisition practitioners will come to accept the premise that a product line perspective is the best-value approach for addressing the needs of most missions. The goals listed in Table 2 further characterize how this vision can be achieved.
Table 2: Long-Term Goals for Product Lines in Acquisition
Achieving the long-term vision for product lines in DoD acquisition will require a broad systematic effort to educate and support acquisition and development practitioners.
These actions are consistent with the mission and current plans of the SEI's Product Line Practice Initiative.
All of the institutions that have a stake in acquisition can become more effective in their respective roles by taking a product line perspective.
References
[Bachmann 00]
Copyright 2002 by Carnegie Mellon University.
NO WARRANTY
THIS CARNEGIE MELLON UNIVERSITY AND SOFTWARE ENGINEERING INSTITUTE
MATERIAL IS FURNISHED ON AN "AS-IS" BASIS. CARNEGIE MELLON UNIVERSITY MAKES NO
WARRANTIES OF ANY KIND, EITHER EXPRESSED OR IMPLIED, AS TO ANY MATTER
INCLUDING, BUT NOT LIMITED TO, WARRANTY OF FITNESS FOR PURPOSE OR
MERCHANTABILITY, EXCLUSIVITY, OR RESULTS OBTAINED FROM USE OF THE MATERIAL.
CARNEGIE MELLON UNIVERSITY DOES NOT MAKE ANY WARRANTY OF ANY KIND WITH RESPECT
TO FREEDOM FROM PATENT, TRADEMARK, OR COPYRIGHT INFRINGEMENT.
Use of any trademarks in this report is not intended in any way to infringe on the rights of the trademark holder.
Internal use. Permission to reproduce this document and to prepare derivative works from this document for internal use is granted, provided the copyright and "No Warranty" statements are included with all reproductions and derivative works.
External use. Requests for permission to reproduce this document or prepare derivative works of this document for external and commercial use should be addressed to the SEI Licensing Agent.
This work was created in the performance of Federal Government Contract Number F19628-00-C-0003 with Carnegie Mellon University for the operation of the Software Engineering Institute, a federally funded research and development center. The Government of the United States has a royalty-free government-purpose license to use, duplicate, or disclose the work, in whole or in part and in any manner, and to have or permit others to do so, for government purposes pursuant to the copyright license under the clause at 252.227-7013.
For information about purchasing paper copies of SEI reports, please visit the
publications portion of our Web site (http://www.sei.cmu.edu/publications/pubweb.html).
REPORT DOCUMENTATION PAGE
Public reporting burden for this collection of information is estimated to average 1 hour per response, including the time for reviewing instructions, searching existing data sources, gathering and maintaining the data needed, and completing and reviewing the collection of information. Send comments regarding this burden estimate or any other aspect of this collection of information, including suggestions for reducing this burden, to Washington Headquarters Services, Directorate for information Operations and Reports, 1215 Jefferson Davis Highway, Suite 1204, Arlington, VA 22202-4302, and to the Office of Management and Budget, Paperwork Reduction Project (0704-0188), Washington, DC 20503. 1. agency use only (leave blank)
2. report date
June 2002
3. report type and dates covered
Final
4. title and subtitle
A Software Product Line Vision for Defense Acquisition
5. funding numbers
C — F19628-00-C-0003
6. author(s)
Grady H. Campbell, Jr.
7. performing organization name(s) and address(es)
Software Engineering Institute
8. performing organization
CMU/SEI-2002-TN-002
9. sponsoring/monitoring agency name(s) and address(es)
HQ ESC/XPK
10. sponsoring/monitoring
11. supplementary notes
12.a distribution/availability statement
Unclassified/Unlimited, DTIC, NTIS
12.b distribution code 13. abstract (maximum 200 words)
Experience in industry and government over the last 10 years has shown that a
software product line approach can significantly improve productivity and
product quality, facilitate change, and reduce life-cycle costs. Defense
acquisition policy calls for such improvements related to software but makes
no explicit mention of a product line approach as an option. Although policy
gives program managers sufficient flexibility to adopt a product line
approach, there is little awareness of this possibility and considerable
uncertainty concerning when and how to do it. This note presents a vision for
software product lines as an acquisition focus and suggests extensions to
current Department of Defense policy and practices to increase the awareness
of and receptivity to product line acquisition as a viable alternative.
14. subject terms
software, acquisition, product line, life cycle
acquisition policy
15. number of pages
16. Price Code
17. security classification
UNCLASSIFIED
18. security classification
UNCLASSIFIED
19. security classification
UNCLASSIFIED
20. limitation of abstract
UL
NSN 7540-01-280-5500
Production and Deployment Phase
Transition Actions for the Long-Term Vision
Goal
Enhancement
Rationale
LV.1
Every acquisition program will be characterized from a product line perspective.
To achieve equal standing between point-solution and product line approaches, each program must make an explicit determination as to which approach is most suitable for satisfying its mission needs. An analysis of possible variabilities and uncertainties in mission needs is necessary either to justify a product line approach or to justify that a point solution will suffice.
LV.2
Every acquisition program that would benefit from a product line perspective in addressing the targeted life-cycle needs of a mission will choose to adopt a product line approach. Point solutions will be taken only as part of an interim scale-up phase or when justified based on specific economic or feasibility criteria.
When the benefits of a product line approach to DoD acquisition have been broadly established, the burden will shift from justifying a product line perspective to justifying a point-solution perspective. Nevertheless, there may always be special situations in which a point-solution approach will be preferred.
LV.3
Suppliers will be qualified as bidders based on evidence of existing, capitalized product line capabilities that are applicable to the targeted life-cycle needs of a DoD mission.
The full benefits of a product line approach to acquisition are achievable only if suppliers also take a product line perspective in their businesses and build capabilities that respond to future DoD needs.
Bachmann, F.; Bass, L.; Chastek, G.; Donohoe, P.; & Peruzzi, F. The
Architecture Based Design Method (CMU/SEI-2000-TR-001, ADA375851). Pittsburgh,
PA: Software Engineering Institute, Carnegie Mellon University, 2000.
[Bergey 99]
Bergey, J.; Fisher, M.; & Jones, L. The
DoD Acquisition Environment and Software Product Lines (CMU/SEI-99-TN-004,
ADA244787). Pittsburgh, PA: Software Engineering Institute, Carnegie Mellon
University, 1999.
[Bergey 01]
Bergey, J.; Campbell, G.; Cohen, S.; Fisher, M.; Gallagher, B.; Jones, L.;
Northrop, L.; & Soule, A. Software
Product Line Acquisition¾A Companion to
A Framework for Software Product Line Practice, Version 1.0. Pittsburgh,
PA: Software Engineering Institute, Carnegie Mellon University, 2001.
[Chastek 01]
Chastek, G.; Donohoe, P.; Kang, K.; & Thiel, S. Product
Line Analysis: A Practical Introduction (CMU/SEI-2001-TR-001, ADA396137).
Pittsburgh, PA: Software Engineering Institute, Carnegie Mellon University,
1996.
[Clements 00]
Clements, P. & Northrop, L. A Framework for Software Product
Line Practice, Version 3.0. Pittsburgh, PA: Software Engineering Institute,
Carnegie Mellon University, 2000. .
[Clements 01]
Clements, P.; Kazman, R.; & Klein, M. Evaluating Software Architectures:
Methods and Case Studies. Boston, MA: Addison-Wesley, 2001.
[Clements 02]
Clements, P. & Northrop, L. Software Product Lines - Practices and Patterns.
Boston, MA: Addison-Wesley, 2002.
[Cooper 99]
Cooper, J.; Fisher, M.; & Sherer, S. W. Software
Acquisition Capability Maturity Model (SA-CMM), Version 1.02 (CMU/SEI-99-TR-002,
ADA362667). Pittsburgh, PA: Software Engineering Institute, Carnegie Mellon
University, 1999.
[DoD 00]
United States Department of Defense. The Defense Acquisition System, Directive 5000.1
Washington, D.C.: Office of the Deputy Secretary of Defense, October 23,
2000.
[DoD 01]
United States Department of Defense.
Operation of the Defense Acquisition System, Instruction 5000.2 Washington,
D.C.: Office of the Under Secretary of Defense (Acquisition, Technology,
and Logistics), January 4, 2001.
[Donohoe 00]
Donohoe, P., ed. Software Product Lines: Experience and Research Directions.
Denver, CO, August 28-31. Boston, MA: Kluwer Academic Publishers, 2000.
[SEI]
Software Engineering Institute.
1
Note that the DoD documents referenced here are recent revisions that
differ significantly from past versions, including those referenced by
Bergey, Fisher, and Jones.
The Software Engineering Institute is a federally funded research and development center sponsored by the U.S. Department of Defense.
Form Approved
OMB No. 0704-0188
Carnegie Mellon University
Pittsburgh, PA 15213
report number
5 Eglin Street
Hanscom AFB, MA 01731-2116
agency report number
48
of report
of this page
of abstract
Standard Form 298 (Rev. 2-89)
Prescribed by ANSI Std. Z39-18 298-102