Use of Quality Attribute Workshops (QAWs) in Source Selection for a DoD System Acquisition: A Case Study

John K. Bergey
William G. Wood

CMU/SEI-2002-TN-013
June 2002

Architecture Tradeoff Analysis Initiative

Unlimited distribution subject to the copyright.

[Top] [Abstract] [About this Series] [1 Introduction] [2 DoD System Acquisition Context] [3 Quality Attribute Workshop (QAW)] [4 The Acquisition Strategy of the MSIS] [5 Integrating the QAW in the MSIS Acquisition] [6 Accomplishments, Lessons Learned, and Prognosis] [7 Summary] [Feedback and Contact] [A Example QAW Scenario and Architectural Test Case] [B Example Rules of Engagement for the QAW] [C Section H of the RFP: Special QAW Requirements] [D Section L of the RFP: Offerors Instructions for the QAW] [E Section M of the RFP: QAW Technical Evaluation Criteria]
[References] [PDF File]

About the Technical Note Series on Architecture Evaluation in the Department of Defense

The Product Line Systems Program is publishing a series of technical notes designed to condense knowledge about architecture evaluation practices into a concise and usable form for Department of Defense (DoD) acquisition managers and practitioners. This series is a companion to the Software Engineering Institute (SEI) series on product line acquisition and business practices.

Each technical note in the series focuses on the use of architecture evaluation and, in particular, on applying the SEI's architecture tradeoff analysis technology in the DoD and government organizations. Our objective is to provide practical guidance on how those organizations can integrate sound architecture evaluation practices into their acquisitions. This series of technical notes lays down a conceptual foundation for DoD architecture evaluation practice.

1 Introduction

This technical note is a case study of how a Department of Defense (DoD) organization is applying architecture analysis and evaluation in a system acquisition, early in the source-selection process to reduce program risk.

In Section 1, Section 2, and Section 3 of the case study, we describe the system being acquired, the motivation for including architecture analysis and evaluation in the acquisition, and the Quality Attribute Workshop (QAW) approach. Following this, we provide a brief description of the system acquisition strategy (Section 4) and the set of events and supporting artifacts that were required to incorporate QAW architecture analysis in the acquisition strategy (Section 5). The relationship of these events and artifacts to the source-selection process is also described. Concluding the case study is a description of the accomplishments and lessons learned (Section 6), along with sample sections from the request for proposal (RFP). These sections provide additional insight into the contractual language that was used to implement the architecture analysis and evaluation approach.

In a software-intensive system, the architecture of the system significantly influences the performance and other qualities of that system. The early use of architecture analysis and evaluation can help mitigate many of the risks associated with system development, thereby improving the ability of an organization to achieve its stated system objectives.¹ In an acquisition context, these analyses and evaluations provide the acquirer with a proactive means of

gaining early visibility into critical design decisions that will drive the entire system and software development effort

determining if a system being proposed by a supplier will satisfy its desired system quality attributes (for example, performance, availability, and modifiability) before the system and software are actually built

This is different from an architectural review that is a typical part of an acquisition milestone, such as a Critical Design Review. These architectural reviews are relatively superficial and rarely address the software architecture² of the system.

1.1 Terminology

Source selection encompasses the process of evaluating offerors (i.e., bidders) in accordance with the evaluation factors specified in the governing RFP and awarding one or more contracts. It is often referred to as a "down select" because it typically results in the elimination of many offerors from consideration and the selection of a small number (e.g., one or two) for a contract award.

In the DoD acquisition environment, the term evaluation has special meaning because evaluation is an integral part of the source-selection process that is common to all government acquisitions. What is commonly referred to as an architecture evaluation in the technical arena will be referred to as an architecture analysis in this technical note. We will reserve the use of the word evaluation in reference to the source-selection process.³ In this context then, conducting an architecture analysis and evaluation means analyzing an architecture (and producing a report on the analysis results) and evaluating the analysis results in strict accordance with the technical evaluation criteria of Section M of the governing RFP.

2 DoD System Acquisition Context

This case study involves an ongoing DoD acquisition. The identities of the DoD organization and system have been purposefully disguised because the acquisition is in a sensitive phase. We will refer to the DoD acquisition organization as the DAC and the system they are acquiring as the MSIS-a maintenance support information system.

2.1 The System Being Acquired: MSIS

The MSIS is a complex system of systems that consists of three major nodes encompassing command, regional, and local maintenance centers. These centers support the maintenance of weapons platforms that are operationally deployed. Maintaining equipment for these platforms at the local centers is performed by the operational units and is largely unscheduled. Maintenance at the regional centers is performed on platforms sent to the regional centers for scheduled overhauls. Maintenance at the command centers, on the other hand, focuses on maintenance planning and analysis activities. From a total system perspective, all the centers have areas of overlapping operations and functionality. Figure 1 provides an overview of the MSIS concept.

Figure 1. Conceptual View of the MSIS

The MSIS will have to provide timely information to support the maintenance needs of a wide variety of DoD operational weapons platforms at geographically dispersed sites. The MSIS's mission is to provide the support needed by the command, regional, and local centers to plan, coordinate, schedule, and monitor the repair, maintenance, and upgrade of these weapons platforms, or their overhaul and refurbishment. For each platform, the system maintains its configuration and the part number of each of its elements. The system also requisitions parts from inventory and maintains the data associated with the deficiencies being repaired by the maintainers and the required maintenance equipment and facilities. The ability to communicate reliably with these centers and rapidly access their maintenance information system databases will be a major element in managing and rendering the maintenance services needed to support the MSIS's customers. As a result, the system specification for the MSIS emphasizes not only functionality but also specific quality attributes (e.g., performance, interoperability, availability, security, usability, and modifiability) that reflect these needed system capabilities. Of course, the DAC must also consider the functional requirements associated with performing the MSIS's mission, but that is not the focus of this technical note. If functionality were all that mattered, any monolithic architecture would do, but other things¾namely the quality attributes¾also matter. Throughout the development process, the architecture must play a role that is both prescriptive and descriptive. Even in an incremental or spiral approach, the core architectural decisions that support the important system quality attributes must come first, and then they can be enhanced in future increments or spirals. An architecture-centric approach, though, is key to the development of systems to meet both their functionality and quality goals.

2.2 Motivation for Incorporating Architecture Analysis

The architecture is the foundation for any system. It represents the earliest design decisions that are both the most difficult to get right and the hardest to change downstream. The architecture will allow or preclude nearly all of the system's qualities. Modifiability, performance predictability, security, availability, interoperability, and usability are all precast when the architecture has been established. No amount of later tuning and implementation tactics will compensate for the ills of a poorly constructed architecture. Experience has shown that an unsuitable architecture will eventually precipitate some sort of disaster on a project [Kazman 00], [Clements 02a]. That disaster may mean failure to meet the performance goals, failure to interoperate as needed, and/or inordinate sustainment costs, among others. It follows then that the design of a system's architecture is key to achieving the system's goals.

As a result, the ability to analyze and evaluate architectures early in the acquisition cycle can help ensure that the delivered systems will meet these goals. This was a major driver for incorporating architecture analysis in the MSIS acquisition and including it as a major evaluation factor in the source-selection process.

One of the challenges facing the DAC was determining what would be required to incorporate architecture analysis and evaluation in its acquisition so it could analyze a contractor's proposed design to see if it satisfied the system's quality requirements.

3 Quality Attribute Workshop (QAW)

The DAC chose to use the Software Engineering Institute's (SEI's) Quality Attribute Workshop (QAW) [Barbacci 02] as its architecture analysis "method of choice." The QAW is based on techniques successfully applied in the Architecture Tradeoff Analysis Method^SM (ATAM^SM) [Kazman 00]. The purpose of the ATAM is to assess the consequences of architectural decisions in light of quality attribute requirements and business drivers. Not only can it analyze specific architecture quality attributes, but it also allows engineering tradeoffs to be made among possibly conflicting system quality goals. In this way, an ATAM evaluation can detect areas of potential risk within the software architecture of a complex software-intensive system. Clements and associates provide details and uses of the ATAM (and other software architecture analysis methods) [Clements 02a]. The software architecture must be documented before the ATAM can be conducted. One of the steps of the ATAM involves facilitating a group of system stakeholders to brainstorm and prioritize scenarios that characterize required quality attributes. The architecture is then analyzed against these scenarios. The results of an ATAM evaluation include a listing of risks, tradeoffs, and sensitivity points.

In the QAW, the quality attributes and business drivers are also established early, and the same technique that is used in ATAM is used to brainstorm and prioritize scenarios. These scenarios are then turned into architectural test cases (ATCs), which also include questions concerning the architectural issues related to the desired quality attributes and suggestions for how to respond to each question. The system developer is responsible for building the architecture and performing the entire analysis of the architecture offline, while the role of the architecture evaluation team is to review and evaluate the analysis results. The QAW method is an outgrowth of SEI work with DoD customers who wanted to apply architecture analysis and evaluation early in the acquisition cycle. The method itself is still evolving as we learn to wrestle with the problem of applying architecture analysis before a software architecture has been fully crafted.

In the QAW, unlike in the ATAM, the architecture need not be available at the early stages when the scenarios are generated and the ATCs are built. However, at least a partial architecture must be available before the analysis of the test cases can proceed. The architecture must be sufficiently detailed to satisfy the "expected response" for each "question" in each ATC, and hence will include elements of both a system architecture [AWG 98] and a software architecture [Clements 02a]. It is fully expected that the ATCs will drive the early architecture view development, since the questions define which parts of the architecture will be analyzed, and the expected responses indicate the level of detail necessary in the analysis. Hence, the sequence diagrams built in the early stages will be those that are called out in an "expected response" section. This is appropriate, since the ATCs represent high-priority operational cases that strongly correspond to the project's business drivers.

Like the ATAM, the QAW helps to uncover risks related to architectural decisions that might create future problems with regard to a quality attribute goal. Discovered risks can then be evaluated and made the focus of mitigation activities (e.g., further design, analysis, or prototyping). The QAW provides an early reasoning framework that can guide system development to help ensure that quality goals are achieved.

3.1 QAW Process

In this section, we provide a brief overview of the QAW process and leave it to the readers to familiarize themselves with all the technical details of the QAW process [Barbacci 02]. The QAW process consists of four distinct groups of activities shown in Figure 2. These activities include: (1) Scenario Generation, (2) Test Case Development, (3) Test Case Architecture Analysis, and (4) Analysis Results Presentation.

Figure 2. Overview of the QAW Process

3.1.1 Scenario Generation

A scenario is a short statement describing an interaction of one or more stakeholders with the system [Clements 02a]. Scenarios are used to represent stakeholders' interests and quality attribute requirements and to exercise the architecture against current and future situations. During scenario generation, individual stakeholders, in a round-robin brainstorming fashion, propose scenarios or ask questions about the way in which the architecture will respond to various situations. The stakeholders who typically take part in this QAW activity include domain experts, technology experts, maintainers, and users. The output of this activity is a prioritized set of scenarios with an additional refinement of the highest priority scenarios (typically 3 to 5). Appendix A contains an example of a typical MSIS scenario.

3.1.2 Test Case Development

The same stakeholders who participate in scenario generation are usually involved in test case development. Test case development transforms each refined scenario into a well-documented test case. A test case is a fully developed, robust scenario that includes

a context section that outlines the operational conditions that form the basis for the test case

a series of questions stating the corresponding architectural issues and concerns

an expected response (for each question) that suggests how the architecture development team should respond to each question

a utility table⁴ that summarizes the quality attributes, the particular system aspects being addressed, and their relationship to the questions

Appendix A contains an example ATC for MSIS and its corresponding scenario.

3.1.3 Test Case Architecture Analysis

In the Test Case Architecture Analysis activity, the architecture development team independently analyzes the architecture using the ATCs and documents the results. An appropriate set of ATCs and architecture documentation are prerequisites for performing this analysis. Because there are no generally accepted, industry-wide standards for describing architectures, the analyses are often constrained by the available documentation. In the case of systems documented using the Command, Control, Communications, Computer, Intelligence, Surveillance and Reconnaissance (C4ISR) Architecture Framework⁵ [AWG 98], different products or collections of products will differ in their relative value for analyzing quality-attribute-specific ATCs [Barbacci 99]. Depending on the quality attributes of concern, C4ISR products will have to be augmented with additional architectural views and documentation to address quality attribute concerns that are under-represented in the C4ISR products. For example, the architecture development team may need sequence diagrams showing the behavior of the major system components and the sequences of messages passing between them. Clements and associates provide practical guidance on how to capture an architecture in written form so it can fulfill its purpose as a communication vehicle for all the varied stakeholders in a system's development [Clements 02b].

3.1.4 Analysis Results Presentation

The Analysis Results Presentation is the final activity in the QAW process. It is a one- or two-day meeting attended by the architecture evaluation team and the architecture development team. It provides an opportunity for the architecture development team to present the results of its analysis and to demonstrate that the proposed architecture can handle the ATCs correctly. A workbook containing a summary of the QAW process, the collection of ATCs, and example analyses and results presentations is provided to the architecture development team in advance. The workbook contains example artifacts that are useful for guiding the team through each QAW activity.

3.2 Applying the QAW Process

Using the QAW process in a system acquisition context requires careful planning that involves appropriately tailoring and integrating the four QAW activities in a manner that is fully compatible with the needs of the acquiring organization. For the MSIS, the starting point was to understand the DAC's acquisition strategy. This is the focus of the next section.

4 The Acquisition Strategy of the MSIS

The acquirer's acquisition strategy defines its overall approach for how it intends to acquire a system. Figure 3 depicts the five phases in the overarching acquisition strategy the DAC selected for the MSIS acquisition. A substantial Planning and Preparation phase (during which the acquisition strategy was developed) preceded these five phases.

Figure 3. Overview of the DAC's Acquisition Strategy for Acquiring the MSIS

The strategy begins with an "open" or competitive solicitation, followed by a source selection that leads to an initial down select. This initial down select results in contract awards to two suppliers to participate in a competitive system development commonly referred to as a "fly-off." The selected suppliers (contractors A and B) subsequently begin the Competitive Fly-Off¾the initial performance phase of their MSIS development effort¾in accordance with their technical proposal/contract. Near the end of the fly-off, the DAC issues a Call for Improvement (CFI) requesting each contractor to submit a revised technical proposal that incorporates the results of the work they performed and the understanding they gained during the competitive fly-off. In the final down select, the acquirer evaluates each contractor's improved technical proposals, and awards a contract for system implementation to the supplier submitting the "best value" proposal.

An important aspect of the acquirer's acquisition strategy is that it sets the context for incorporating architecture analysis and evaluation in an acquisition. The next section discusses the planning and preparation that went into integrating the QAW with the MSIS acquisition strategy.

5 Integrating the QAW in the MSIS Acquisition

The acquisition phases (depicted in Figure 4) serve as a roadmap to show how the SEI assisted the DAC in incorporating the QAW into its acquisition. This roadmap corresponds to the initial acquisition Planning and Preparation phase followed by the five phases of the acquisition strategy.

Figure 4. MSIS Acquisition Roadmap

The subsections that follow correspond one to one with these six phases and include tables that describe the specific events and supporting artifacts that were required to create the acquisition infrastructure. This infrastructure accommodates the QAW process and provides a suitable means for evaluating the QAW architecture analysis results. Table 1 provides the legend for understanding the allocation of roles and responsibilities for the numbered events described in the other tables in this section.

Table 1. Legend for Identifying Roles and Responsibilities for QAW Events

Acquisition Phase

BLACK FILL around the event number means that the DAC is responsible for performing all the tasks for that particular event.

2 .

DARK GRAY FILL around the event number means the DAC and the contractor are jointly responsible for conducting the event in a collaborative manner (in accordance with the governing plan).

Any task that is SHADED IN LIGHT GRAY in the joint event is performed by the DAC.

Any task that is NOT SHADED in the joint event is performed by the contractor.

NO FILL around the event number means that the offeror or contractor is responsible for performing all the tasks for that event (in accordance with the RFP or an approved plan).

5.1 Planning and Preparation

Typical of acquisitions of this magnitude, the MSIS acquisition involved a significant amount of planning and preparation. This work involved establishing the acquisition objectives, developing the detailed acquisition strategy, identifying and managing risks, specifying the system requirements, developing cost and schedule estimates, and preparing the RFP. The DAC was responsible for executing and coordinating all of these activities.

A major way in which the DAC elected to reduce overall program risk was to incorporate QAW-based architecture analysis and evaluation and to integrate it into the DAC's source-selection process. This required additional planning and preparation. Table 2 describes the acquisition Planning and Preparation phase events that are relevant to integrating the QAW in the MSIS acquisition.

Table 2. Acquisition Planning and Preparation-QAW-Related Events

PHASE	EVENT	Description of Event and Relevance to QAW
PLANNING AND PREPARATION	1.	DAC management establishes the objectives and governing policies and parameters for the MSIS acquisition. Some of these affect how the QAW was integrated into the acquisition. Examples include mandating compliance with the prescribed acquisition schedule mandating the use of a Statement of Objectives (SOO) approach for structuring the RFP instead of a traditional Statement of Work (SOW) requiring some aspect of the QAW to serve as a “discriminator” in source selection for both the initial and final down select
	2.	The DAC forms a team to plan the acquisition, develop an acquisition strategy, and prepare the RFP. In conjunction with the DAC, SEI team members identify the business drivers and system quality attributes that are important and that reflect the business drivers plan and carry out the first two QAW activities—Scenario Generation and Architecture Test Case Development—in parallel with the other acquisition planning and preparation efforts coordinate the specification of quality attribute requirements coordinate the specification of the architectural documentation requirements (e.g., augmented C4ISR views) for analyzing the architecture’s ability to support the quality attribute requirements specify appropriate acquisition artifacts and develop corresponding language for the SOO and Sections H, L, and M of the RFP to integrate the remaining QAW activities—Test Case Architecture Analysis and Analysis Results Presentation—consistent with the acquisition events described in these tables
	3.	The DAC completes the RFP and obtains management approval to begin the solicitation: included in the RFP is a set of robust ATCs for use in the QAW Test Case Architecture Analysis and Analysis Results Presentation activities

As part of the acquisition Planning and Preparation phase, SEI team members were responsible for carrying out the first two QAW activities¾Scenario Generation and Test Case Development. This involved planning and facilitating four separate working sessions with MSIS stakeholders at three different sites over a period of several months. These sessions included stakeholders from the command, regional, and local centers. The stakeholders included acquirers, domain experts, and prospective users of the MSIS.

As part of performing these two QAW activities, the DAC/SEI team identified

the business drivers for the MSIS

the system quality attributes that are important and that reflect the business drivers

architectural documentation (e.g., augmented C4ISR views) that are needed to permit an analysis of the architecture's ability to support the desired quality attributes

The end goal was to quickly develop a sufficient number of ATCs in time to be included in the RFP. These ATCs reflected the quality attributes that were essential to conducting a core set of maintenance tasks at the command, regional, and local centers. In developing these ATCs, there was a strong emphasis on the processes and personnel used to perform these maintenance tasks over a long period of time, which required extensive context setting.

All together, eleven ATCs were generated for inclusion in the RFP. They will drive the final two QAW activities¾Test Case Architecture Analysis and Analysis Results Presentation. Performing these remaining two QAW activities is the responsibility of the contractors who win the MSIS competitive fly-off contracts. How the MSIS contractors are expected to conduct these activities (in conjunction with the DAC) is described in Section 5.4.

5.2 Competitive Solicitation

The second acquisition phase is the Competitive Solicitation that encompasses three events relevant to integrating the QAW in the MSIS acquisition. Table 3 describes them.

Table 3. Competitive Solicitation-QAW-Related Events

PHASE	EVENT	Description of Event and Relevance to QAW
COMPETITIVE SOLICITATION	1.	The DAC conducts bidders’ conferences to inform potential offerors about the particulars of the upcoming MSIS acquisition. The SEI in conjunction with the DAC presents briefing on the QAW process to familiarize offerors with the architecture analysis approach and acquisition implications.
	2.	The DAC releases the RFP: ATCs are included in the RFP as government-furnished items (GFIs) so that offerors can gauge the scope of the required analysis. QAW workbooks are also included in the RFP to serve as a guide for planning and conducting the QAW Test Case Architecture Analysis and Analysis Results Presentation activities.
	3.	Offerors deliver technical proposals in accordance with Section L of the RFP: Section L requires that the technical proposals include an initial architecture analysis plan (AAP).

A key aspect of this phase is requiring each offeror⁶ to submit an initial architecture analysis plan (AAP) as part of its technical proposal. The AAP describes how the offeror intends to conduct QAW-based architecture analysis if it is awarded a contract to participate in the competitive fly-off. The scope of the plan is to include both the QAW Test Case Architecture Analysis and Analysis Results Presentation activities. Since each offeror is required to submit an initial plan, the DAC will have a tangible means for evaluating each offeror's ability to

understand the QAW process and plan a QAW-based architecture analysis

make reasonable assumptions in applying the QAW process and integrate it appropriately into the offeror's proposed development effort

satisfy the technical proposal requirements of the RFP that pertain to QAW architecture analysis

This approach allows architecture analysis to be a factor in the initial source selection and greatly reduces the risk of selecting an offeror that will be unable to follow through successfully.

5.3 Initial Down Select

The third acquisition phase is the Initial Down Select that encompasses three events relevant to integrating the QAW in the MSIS acquisition. Table 4 describes them.

Table 4. Initial Down Select-QAW-Related Events

PHASE	EVENT	Description of Event and Relevance to QAW
INITIAL DOWN SELECT	1.	The DAC evaluates offerors’ technical proposals in accordance with the technical evaluation criteria in Section M of the RFP (for the initial down select) This includes evaluating each offeror’s initial AAP.
	2.	The DAC factors in the results of the technical proposal evaluations in source selection in accordance with the overall weighting criteria in Section M of the RFP (for initial down select): AAP evaluation results are a significant factor in the evaluation of the technical proposals.
	3.	The DAC makes the initial down select and awards contracts to two of the offerors to participate in the competitive fly-off in accordance with the DAC’s source-selection plan.⁷ Each selected offeror will be responsible for conducting QAW activities in accordance with its plan and the RFP/contract requirements.

The DAC anticipated receiving many technical proposals prior to the initial down select and needed an effective means to screen out all but the most highly qualified offerors in order to minimize program risk and make the down select more efficient. Moreover, due to the importance of architecture analysis to the MSIS program, the DAC wanted some aspect of architecture analysis and evaluation to play a major role in source selection¾even in the initial down select. The means of achieving this is to evaluate the efficacy of each offeror's AAP submitted as part of its technical proposal during the initial down select. Otherwise, architecture analysis and evaluation could not play a role in source selection until the final down select when the analysis results from the Competitive Fly-Off phase would come into play.

5.4 Competitive Fly-Off

The last two QAW activities¾Test Case Architecture Analysis and Analysis Results Presentation¾take place in the Competitive Fly-Off phase. The contractors conduct these activities in two segments; doing so benefits both the DAC and the contractors as explained in the following sections. In the first segment, the contractors conduct a "dry-run" Test Case Architecture Analysis and Analysis Results Presentation. In the second segment, they conduct "full-scale" ones. The main difference between these two segments is that the dry-run segment involves the use of only a few designated ATCs, while the full-scale segment requires the use of all the GFI ATCs.

5.4.1 QAW Dry-Run Segment

The dry-run segment encompasses the seven events described in Table 5. The benefit of first having a dry run is that it provides the contractors with an opportunity to familiarize themselves with the QAW process, fine-tune their analysis skills, make any needed architectural changes, and refine their plans for performing the analysis.

5.4.2 QAW Full-Scale Segment

Table 6 describes the five events encompassing the full-scale segment. This segment of the competitive fly-off involves all the ATCs and requires a final report of the analysis results. A unique aspect of this segment is that the DAC will issue a CFI 30 days before the segment is scheduled to end. In response to the CFI, the contractors must prepare an improved technical proposal describing their plans for the final System Implementation phase. Since the final architecture analysis report is required to be part of the improved technical proposal, the analysis results can be included as a major evaluation factor in source selection for the final down select.

Table 5. Competitive Fly-Off-QAW-Related Events for Dry-Run Segment

PHASE	EVENT	Description of Event and Relevance to QAW
First Segment of COMPETITIVE FLY-OFF	1.	The DAC conducts a kick-off briefing with each of the two contractors to review government expectations for the competitive fly-off. The contracting officer reviews the rules of engagement (ROE).⁸ The DAC reviews the requirements governing the fly-off and the SEI presents a briefing on the QAW analysis and evaluation approach.
	2.	Contractors refine their initial AAPs, as necessary, based on the comments they received from the DAC (originating from the initial down select).
	3.	The DAC reviews each contractor’s refined AAP and grants preliminary approval (or cites rework required by the contractor).
	4.	Upon receiving preliminary approval of its refined AAP, each contractor builds sufficiently detailed MSIS architectural views and documentation to enable the architecture analysis for the dry-run test cases performs a dry-run QAW Test Case Architecture Analysis⁹ using designated GFI ATCs and available architectural views prepares a presentation detailing the architecture analysis results
	5.	Each contractor in conjunction with the DAC conducts a dry-run QAW Analysis Results Presentation.¹⁰ Each contractor presents the architecture analysis results for the small set of designated ATCs on a test-case-by-test-case basis. The DAC reviews and discusses the analysis results and the contractor's progress against their refined AAPs. The DAC issues evaluation notices (ENs),¹¹ as necessary, based on its review and technical interchange with the contractors in accordance with the QAW’s ROE.
	6.	Each contractor revises its AAP, as necessary, to incorporate planning changes and refinements, or to accommodate the ENs received from the DAC.
	7.	The DAC reviews each contractor’s revised AAP and grants final approval (or cites rework required by the contractor).

Table 6. Competitive Fly-Off-QAW-Related Events for Full-Scale Segment

PHASE	EVENT	Description of Event and Relevance to QAW
Second Segment of COMPETITIVE FLY-OFF	8.	Upon receiving final approval of its revised AAP, each contractor completes build of all MSIS architectural views and documentation to enable the architecture analysis for the full-scale test cases performs a full-scale QAW Test Case Architecture Analysis using all GFI ATCs and completed architectural views prepares a presentation describing the architecture analysis results
	9.	Each contractor in conjunction with the DAC conducts a full-scale QAW Analysis Results Presentation. Each contractor presents the architecture analysis results for all ATCs on a test-case-by-test-case basis. The DAC reviews (and conducts dialog on) the analysis results and the contractors’ progress, comparing it against their revised AAPs. The DAC issues ENs, as necessary, based on its review and technical interchange with the contractors.
	10.	Each contractor prepares an Architecture Analysis Report (AAR) in accordance with Section L of the RFP. The AAR includes architecture analysis results that were presented during the QAW presentation of the analysis results for all ATCs architectural views and documentation used in the analysis and other supporting information as described in QAW workbooks responses to all ENs issued by the DAC describing either how the ENs were resolved or mitigation plans for resolving them any new or revised architectural documentation corresponding to proposed changes or resolution of ENs received from the DAC
	11..1.	The DAC issues a CFI.
	12.	Contractors deliver their improved technical proposals, in accordance with Section L of the RFP. The improved proposals include their final AAR and AAP a complete description of any new architectural changes being proposed including the supporting rationale and impact on architecture analysis results previously presented by the contractor an improved AAP (IAAP) describing the contractor’s approach for conducting an architecture analysis of each software architecture build (i.e., increment) using the ATAM during the System Implementation phase

5.5 Final Down Select

The Final Down Select phase encompasses five events that are relevant to integrating the QAW. Table 7 describes them.

Table 7. Final Down Select-QAW-Related Events

PHASE	EVENT	Description of Event and Relevance to QAW
FINAL DOWN SELECT	1.	The DAC evaluates the improved technical proposals in accordance with technical evaluation criteria in Section M of the RFP (for the final down select). This includes evaluating the AAR,¹² progress against the AAP, and the IAAP.
	2.	Each contractor presents (as part of its oral presentation to the DAC) a summary of its architecture analysis results (including any subsequent architectural changes and their impact), overall performance and progress compared against its AAP, and its proposed IAAP. The DAC conducts separate discussions with each contractor on its oral presentations in accordance with the ROE.
	3.	The DAC evaluates each contractor’s oral presentation in accordance with the technical evaluation criteria in Section M of the RFP (for the final down select).
	4.	The DAC incorporates the evaluation results (encompassing improved technical proposals and oral presentations) into source selection in accordance with the weighting criteria in Section M of the RFP (for the final down select). The architecture evaluation results are a significant factor in the overall evaluation and selection process.
	5.	The DAC makes the final down select and awards a system implementation contract to one of the two fly-off contractors in accordance with the DAC’s source-selection plan.¹³ The winning contractor is responsible for conducting follow-on software architecture analyses during the System Implementation phase in accordance with its submitted plan and the RFP/contract requirements.

A key aspect of this phase is that the DAC architecture evaluation team has an opportunity to review and discuss the final analysis report as well as the contractor's proposed architectural changes (or plans) for mitigating discovered risks or deficiencies that were cited in the ENs. This takes place during the contractor's oral presentation, which is a traditional element of the source-selection process. The contractors must submit their analysis reports in advance of their oral presentations. This allows the DAC architecture evaluation team ample opportunity to carefully review the reports prior to the oral presentations. These presentations are also a factor in evaluating the performance of the competing contractors.

5.6 System Implementation

The System Implementation phase encompasses two follow-on events to the QAW that are relevant to integrating architecture analysis in the MSIS acquisition. Table 8 describes them.

Table 8. System Implementation Phase¾ATAM-Related Events

PHASE

EVENT

Description of Event and Relevance to QAW

SYSTEM IMPLEMENTATION

The winning contractor refines its System Evolution Plan (SEP)¹⁴ and performs the following activities:

The contractor refines its system architecture¹⁵ in increments corresponding to the architecture’s approved SEP.

The contractor refines its system architecture in increments corresponding to its approved SEP.

The contractor refines and completes the development of its software architecture in increments corresponding to its approved SEP.

The contractor conducts an architecture analysis of each increment of the software architecture using the ATAM in accordance with its approved IAAP.

The DAC evaluates the software architecture analysis results for each increment specified in the IAAP.

The DAC appropriately factors ATAM evaluation results into an incentive award in accordance with specific contract provisions.

A key aspect of this phase is that the implementation contractor will conduct software architecture evaluations using the ATAM. These in-situ architecture evaluations will assist the DAC in evaluating the contractor's performance on an ongoing basis during system development. The results and experience that the contractors gain during the competitive fly-off with the QAW should prepare the way for these follow-on software architecture evaluations. These ATAM evaluations should go more smoothly and provide additional insight into risks created by architectural decisions, find trends that reveal correlations between architectural decisions and predictions of system properties, and surface sensitivity points and tradeoffs so they can be identified and documented explicitly.

5.7 Overview of the Primary QAW Events

Figure 5 provides a graphic summary of the approach and acquisition artifacts used for integrating and applying the QAW from the issuance of the RFP through the initial down select. The numbered items identify the sequence in which the QAW-related events occur and associated artifacts are generated. The lightly shaded ellipses identify the first two QAW activities that the DAC is responsible for performing. And the call-out box (number 5) shows the QAW-related artifact that the offerors are responsible for developing and delivering to the DAC.

Figure 5. QAW Integration Approach from Issuance of the RFP Through Initial Down Select

Similarly, Figure 6 provides a graphic summary of the approach and acquisition artifacts used for integrating and applying the QAW from the award of the competitive fly-off contracts through the final down select. The two darkly shaded ellipses (numbers 8 and 11) identify the QAW activities that the fly-off contractors are responsible for performing. And the two call-out boxes (numbers 7 and 10) show the QAW-related artifacts that the contractors are responsible for developing and delivering to the DAC. Item number 9 shows the ENs the DAC will issue to the contractors, as needed, after the Analysis Results Presentation.

Figure 6. QAW Integration Approach from Competitive Fly-Off Through Final Down Select

5.8 Example RFP Language for the QAW

Tables 3 through 8 provide an overview of the detailed strategy that the DAC wrote into the RFP with the SEI's assistance. Because the acquisition strategy described in this case study is very typical of DoD acquisitions [Barbacci 00], many DoD organizations should be able to adapt the approach. The adaptation would involve analyzing the events (described in the tables), tailoring them to the specific needs of the organization, and developing the corresponding RFP/contract language.

Examples of the QAW RFP/contract wording for the ROE, and Sections H, L, and M of the RFP are included in Appendix B through Appendix E, respectively. The examples provided in the appendices for Section M and L do not cover all aspects of the final down select.

[Top] [About this Series] [1 Introduction] [2 DoD System Acquisition Context] [3 Quality Attribute Workshop (QAW)] [4 The Acquisition Strategy of the MSIS] [5 Integrating the QAW in the MSIS Acquisition] [6 Accomplishments, Lessons Learned, and Prognosis] [7 Summary] [Feedback and Contact] [A Example QAW Scenario and Architectural Test Case] [B Example Rules of Engagement for the QAW] [C Section H of the RFP: Special QAW Requirements] [D Section L of the RFP: Offerors Instructions for the QAW] [E Section M of the RFP: QAW Technical Evaluation Criteria]
[References] [PDF File]

6 Accomplishments, Lessons Learned, and Prognosis

6.1 Accomplishments

With the support and collaboration of key MSIS stakeholders from the command, regional, and local centers, good things happened in planning and executing this acquisition initiative. The contribution of the SEI team was to assist the DAC in planning and devising an approach for integrating a QAW. The results included

identifying the business drivers and quality attributes that are important to stakeholders

developing an approach that was compatible with the DAC's acquisition objectives and MSIS acquisition strategy

conducting scenario-generation working sessions with key MSIS stakeholders during the Planning and Preparation phase

completing and coordinating ATC development and producing a robust set of ATCs for inclusion in the RFP that reflect the quality attributes of interest

incorporating a QAW architecture analysis and evaluation objective in the SOO and developing the ROE to govern the process

developing the special contract requirements (Section H) for conducting a QAW and the technical evaluation criteria (Section M) governing the evaluation of QAW results

developing instructions to guide offerors in preparing the QAW-related aspects of the technical proposal (Section L)

creating presentations to brief potential suppliers about the QAW at the upcoming bidder's conferences as well as DAC management, acquisition officials, and other stakeholders

The Scenario Generation and Architecture Test Case Development activities enabled the DAC to establish a proactive means of working with the organizations that will eventually be using the MSIS. Completing these activities enabled the DAC to

better understand the business drivers for the MSIS and what quality attributes are most important to the stakeholders
gauge the degree to which stakeholders and legacy contractors share a common view of how the MSIS should operate
identify and specify scenarios of concern to the stakeholders and the architectural issues and concerns associated with those scenarios
explore how the C4ISR architectural views should be supplemented to support architecture analysis and evaluation
develop a set of ATCs for use in evaluating the ability of the proposed system to achieve the desired quality attributes

The results established a solid "analysis baseline" that the DAC and MSIS stakeholders can build on during the MSIS System Implementation phase. This will assist them in using the ATAM to evaluate the software architecture's inherent quality attribute sensitivities, tradeoffs, and risks early in the System Implementation phase.

From the DAC's standpoint, the bottom line was that all the participants benefited from the experience they gained in preparing for and integrating the QAW.

6.2 Lessons Learned

During the Planning and Preparation phase, the acquisition strategy changed several times. As a result, the SEI team had to modify the approach for integrating the QAW and rewrite sections of the RFP dealing with architecture analysis and evaluation. Developing a robust acquisition strategy should be one of the highest priority items in the early planning phase because changing the strategy downstream can cause instability and require substantial rework of the RFP.

Although it is too soon to gauge the ultimate benefits of this work, the issues that surfaced helped the DAC to discover problems early in the acquisition Planning and Preparation phase, when the expense of correcting them is substantially less. One example is that the original acquisition strategy did not ensure that the results of the competitive fly-off could be used in the final source selection. Another example is that the original technical evaluation criteria for the RFP addressed only the first down select.

There are a few lessons learned that are relevant to other acquisitions desiring to incorporate architecture analysis and evaluation. Each lesson is described below in the form of a problem statement and resulting lessons learned.

How could the DAC require an offeror to plan on conducting an architecture analysis using a specific method (such as the QAW), when, under the tenets of acquisition reform, acquirers are to avoid telling potential suppliers what to do and how to do it?

Lessons learned: First, acquisition reform allows, and even encourages, acquirers to incorporate risk-reduction measures in an acquisition. As a result, it is legitimate for the DAC to require that architecture analysis and evaluation be incorporated as an acquisition risk-reduction measure, since architecture plays such a major role in determining a system's behavior. Second, the DAC must ensure that there is a "level playing field" when evaluating competing architectures. The DAC would not be able to equitably evaluate the architecture analysis results if each offeror were to propose its own architecture analysis method and make different assumptions about the scope and depth of the analysis and the specific architecture aspects it would evaluate. Specifying a QAW provides a common basis for planning and conducting architecture analysis and evaluation across all offerors.

By policy, the DAC was required to use a SOO instead of a traditional SOW for the MSIS RFP. This means that each offeror is responsible for developing a customized SOW based on the SOO and RFP requirements. The first question this policy raises is "what is an appropriate SOO objective to include in the RFP to accommodate architecture analysis and evaluation consistent with a QAW?"

Lessons learned: The objective used read: "Support the objective of reducing program risk by participating in architecture analysis and evaluation for the life of the MSIS acquisition."

The second question this policy raises is "if each offeror is responsible for preparing the SOW that describes the tasks it will perform, how can the DAC explicitly specify in the RFP that the offeror's approach (and hence its SOW) must include performing an architecture analysis using the QAW approach?"

Lessons learned: Use Section H (Special Contract Requirements) of the RFP and include a common set of requirements to govern the planning and conduct of the architecture analysis and evaluation. Specify the scope of the architecture analysis and the analysis method (e.g., QAW or ATAM) in the Section H requirements.

6.3 Prognosis

As a result of integrating the QAW in the MSIS acquisition, the DAC is confident that it will have an effective means to

gauge a potential supplier's ability to plan and conduct an architecture analysis

ensure that the contractors will perform a thorough architecture analysis commensurate with the architectural views and documentation that are available at the time

ensure that the architecture analysis results will provide an equitable basis for evaluating how, and to what extent, the contractor's proposed architecture can or cannot meet the specified technical requirements and system quality attributes

set the stage for conducting software architecture evaluations downstream using the ATAM

reduce program risk

7 Summary

An architecture analysis and evaluation is one risk mitigation activity that has been shown to have a high payoff. Although conducting an architecture evaluation may appear to be an obvious step, it certainly isn't a routine occurrence in the DoD and government environment.

We have presented a case study showing how the SEI was able to assist one DoD organization in effectively integrating a QAW architecture analysis and evaluation in its system acquisition. Integrating architecture analysis and evaluation is dependent on

the goals and objectives of the acquisition program

the organization's acquisition practices

adapting the approach (in this case, the QAW) to the acquisition strategy

ensuring that essential QAW elements and supporting artifacts are incorporated appropriately into the RFP/contract

We have described the specific acquisition events and supporting artifacts needed to incorporate a QAW in a system acquisition so that architecture analysis can play a major role in source selection. This includes providing samples of the contractual language included in the RFP that provide insight into the technical implementation details. A DoD organization should be able to adapt this architecture analysis and evaluation approach to its specific acquisition needs if it has a similar acquisition strategy with a "rolling" down select.

Feedback and Contact

Comments or suggestions about this document or the series of technical notes on architecture evaluation in the DoD are welcome. We want this series to be responsive to the needs of DoD and government personnel. To that end, comments concerning this technical note, the inclusion of other topics, or any other issues or concerns will be of great value in continuing this series. Comments or suggestions should be sent to

Linda Northrop, Director
Product Line Systems Program
lmn@sei.cmu.edu

Software Engineering Institute
Carnegie Mellon University
Pittsburgh, PA 15213

Appendix A Example QAW Scenario and Architectural Test Case

Example Scenario

In one of the scenario-generation brainstorming sessions, 14 scenarios were generated, and facsimiles of three of them are listed below. The participants then grouped these three scenarios together as representing a single scenario.

The scenarios included

(1) A "Weapons Platform Generation Program" is necessary, which is tailorable for each separate command.

(11) During "contingency generation" a simplified information flow is necessary, which: avoids duplication; is tailorable; and provides read access to users at the command center and regional centers.

(13) Must provide the weapons platform's status, configuration, locks, and parking location to the weapons platform's crew and maintenance supervisor. This information should be available to authorized users in the regional and command centers on a need-to-know basis.

The scenario refinement consists of building the outline of the ATC as shown below. First, it defines the context and the organizations involved, and then lists the questions to be used in analyzing the architecture.

Example QAW ATC

The following example is an ATC that was developed during the QAW Test Case Development activity from the initial scenario described above. The participants at the brainstorming session reviewed the scenarios and created an operational context described below, along with a list of the organizations involved and the quality attributes of interest.

1.1 Context

A unit is tasked to deploy to a forward-operating base (contingency generation, hurricane evacuation, etc.) for an unknown duration. Key leadership has a need to know information such as the current status, the location and mission-capable (MC) rate, the long-range scheduled maintenance, depot inputs/outputs, the current status of the hangar queen, and the available hangar space for aircraft being generated (prepared bombs, fuel, etc.). Information will be used to determine which aircraft will be sent and in what order.

1.2 Quality Attributes of Interest

These were performance, availability, interoperability, security, usability, and modifiability. The utility table shows which quality attributes are related to what questions.

1.3 Questions

How will the system generate the reports (weapons platform status report, fleet health report) for the command center users in less than one hour?

Expected Response. Create a sequence diagram showing the distributed business objects used in generating each report. Then elaborate on this diagram to show the communication delays and the computational times required for each node. The timing estimates should account for waiting times for both normal communication traffic and access to computational nodes.

How can the decision makers gain access to the above information (securely and electronically), without having to use a paper copy from the provider?
Expected Response. Determine alternative ways of accessing the data from various devices, enabling interoperability between devices and servers, and formatting the data for screen presentation and browsing.

How will the system prevent unauthorized access to this information without impeding access by the maintenance users at the local center?
Expected Response. Create a delineation o