Specifying Initial Design Review (IDR) and Final Design Review (FDR) Criteria
Mary Ann Lapham
Technical
Note
CMU/SEI-2006-TN-023
Acquisition Support Program
Unlimited distribution subject to the copyright.
[Abstract] [1
Introduction] [2 IDR and FDR Definitions] [3
Engineering Emphasis] [4
IDR and FDR Criteria]
[5 Conclusion] [Appendix
A IDR Criteria] [Appendix
B FDR Criteria] [Appendix C Acronyms]
[References] [PDF
File]
1 Introduction
Many DoD development programs, such as aircraft development programs, are typically complex and long-lived. Often, these programs are structured to demonstrate significant capability in the form of prototypes prior to Milestone B in the Defense Acquisition Management Framework [DoD 03]. This early technology work accelerates and facilitates the application of mature, advanced technologies to DoD programs. As such, these programs frequently include design reviews that are not present in most other systems acquisition life cycles. These supplemental design reviews are most frequently known as the Initial Design Review (IDR) and the Final Design Review (FDR). The IDR and FDR occur during the Technology Development phase of the Defense Acquisition Management Framework [DoD 03]
Following the aircraft development program example, the timeline shown in Figure 1 denotes a notional aircraft acquisition schedule as it relates to the DoD acquisition life cycle. The figure shows the relationship of the IDR and FDR to the overall milestones and other reviews such as Preliminary Design Review (PDR) and Critical Design Review (CDR). As shown, the IDR and FDR are in the Technology Development (TD) phase of the program. The TD phase is the time frame between Milestone A and Milestone B. The PDR, CDR, and Test Readiness Review (TRR) are in the System Development and Demonstration (SDD) phase of the life cycle. The SDD phase occurs between Milestone B and Milestone C.
Figure 1: Notional DoD 5000 Acquisition Timeline
At the end of the TD phase, it is anticipated that working aircraft prototypes exist for evaluation. These prototypes most likely include all segments of the program (radar, platform, munitions, etc.). Given that the prototypes contain a good portion of the capabilities for the aircraft, the IDR and FDR should ensure the required capability is well represented during the demonstration of the prototype and that high-risk items have been addressed to mitigate the risk as much as possible at this time. In addition, the IDR and FDR checkpoints help fulfill the need for increasingly rigorous architectural reviews of system-level capabilities to verify desired emergent properties of the system such as
- security
- flexibility
- extensibility
- configurability
- interoperability
- and others, as identified for the particular system
However, the IDR and FDR content is not explicitly defined in any regulation or policy; rather, they are defined by the program office. Thus, there is a dilemma for the program office. With no guidelines to follow, the program office needs to grapple with what should be included in the IDR and FDR criteria. The remainder of this technical note provides a definition for IDR and FDR, some generic criteria, and some suggestions on how to apply the criteria.
The audiences for this technical note are managers and developers of medium to large DoD systems that employ technologies that are not mature enough to transition directly to systems development.
[Abstract] [1
Introduction] [2 IDR and FDR Definitions] [3
Engineering Emphasis] [4
IDR and FDR Criteria]
[5 Conclusion] [Appendix
A IDR Criteria] [Appendix
B FDR Criteria] [Appendix C Acronyms]
[References] [PDF
File]
2 IDR and FDR Definitions
No specific regulation or policy defines an IDR or FDR; rather, the guidance states that these reviews are defined by the program office. IDR and FDR do fulfill a roughly equivalent purpose in the TD phase as the PDR and CDR do in the SDD phase by providing a way to
- evaluate developed system and software architecture and design
- ensure that desired and required capabilities are well represented during prototype demonstration
- ensure high-risk items have been addressed to mitigate the risk as much as possible
- document engineering and management decisions
Given this purpose, the following are working definitions for IDR and FDR:
IDR - formal technical review of prototype design approach for a configuration item (CI) or computer software configuration item (CSCI) including evaluation of progress, technical adequacy, and risk resolution on a technical, cost, and schedule basis.
FDR - formal technical review of prototype detailed design approach for a CI or CSCI including evaluation of progress, technical adequacy, and risk resolution on a technical, cost, and schedule basis.
Note that the only difference between these definitions is the addition of one word in the FDR definition—detailed. In this instance, "detailed" means that the design contains sufficient information that it could be handed over to a separate development team that could successfully complete the work. While these definitions apply to both hardware and software components of a system, this technical note only discusses CSCIs.
PDR and CDR content is explicitly defined in MIL-STD-1521B. While MIL-STD-1521B, Military Standard Technical Reviews and Audits for Systems, Equipments, and Computer Software, was cancelled April 10, 1995, it is still used today to provide guidance for PDR and CDR technical reviews [DoD 85]. The functional configuration audit and physical configuration audit requirements originally included in MIL-STD-1521B were incorporated into MIL-STD-973, Configuration Management [DoD 92], but all other areas of MIL-STD-1521B were cancelled and not replaced. MIL-STD-973 was cancelled September 30, 2000.
As mentioned earlier, since the IDR and FDR fulfill a roughly equivalent purpose for the TD phase as the PDR and CDR do for the SDD phase, MIL-STD-1521B could also form the basis for IDR and FDR criteria. Of course, the criteria would need to be tailored to meet the needs of the specific program's TD phase.
The IDR and FDR criteria described in this technical note are based on MIL-STD-1521B. In addition, the U.S. Air Force's Software Management Guidebook provides some additional guidance that was used to create the IDR and FDR criteria (in particular, the entrance and exit criteria) presented here [USAF 04].
A few words of caution are in order. MIL-STD-1521B assumed a waterfall approach to software development and functional decomposition of requirements. Today's development environments often incorporate an incremental or spiral development approach, which can affect how the IDR and FDR criteria are applied.
The waterfall approach consists of the requirements development, design, build, unit test, system integration and test, and field steps. In the waterfall method, one step is completed before continuing to the next step. Attributes of the waterfall approach include well-defined, low volatility requirements, a single development cycle, a small and precedented system, and no distribution of interim software. Typically, the hardware and software both follow this cycle, merging at the system integration and test phase.
In the incremental development approach, there can be several cycles of the design, build, and test steps and there is feedback from one cycle to the next. Some of the increments may or may not be fielded. The hardware and software cycles can use different development approaches (i.e., hardware could use the waterfall approach with software using incremental). The attributes of the incremental approach include well-defined, low to moderate volatility requirements, multiple development cycles, any size and possibly unprecedented system, and distribution of interim software.
Finally, the spiral approach is a risk-driven development model. The spiral approach uses cyclic concurrent engineering for which the process and product are determined by risk. This approach grows a system via risk-driven experimentation and elaboration, thus lowering development cost by early elimination of nonviable alternatives and avoiding rework. The attributes of the spiral approach include not well-defined, moderate to high volatility requirements, multiple development cycles, any size and possibly unprecedented system, and distribution of interim software.
Table 1 summarizes the characteristics of each development approach.
Table 1: Characteristics of Development Approaches
With the incremental or spiral approach, a complete functional decomposition of the requirements does not necessarily occur at the beginning of the program; rather, it occurs as the increments are defined or as the program evolves during the spirals. When using the incremental or spiral development approach, there will be multiple IDRs and FDRs, a set occurring for each increment or spiral. Figure 2 shows an example of how the incremental and spiral approaches can be combined during development of a system and where the IDRs and FDRs fit in. A combination of approaches may be used when most of the increments are well defined, but for example, a spiral approach is needed for a particular portion of the system due to risk, lack of technology maturity, or volatile requirements. When using the IDR and FDR criteria proposed in this technical note, keep in mind that the program must take the development environment into account.
Figure 2: Example Incremental/Spiral Approach
[Abstract] [1
Introduction] [2 IDR and FDR Definitions] [3
Engineering Emphasis] [4
IDR and FDR Criteria]
[5 Conclusion] [Appendix
A IDR Criteria] [Appendix
B FDR Criteria]
[Appendix C Acronyms] [References] [PDF
File]
3 Engineering Emphasis
During our review of MIL-STD-1521B and the U.S. Air Force's Software Management Guidebook, we identified 12 software development areas that form the basis of the IDR and FDR content:
- Preliminary Design
- Detailed Design
- Databases
- Interfaces (internal and external)
- Security (information assurance specific to software)
- Control Functions (high-level description of executive control and start/recovery features)
- Resource Allocation (allocation across all CSCIs including timing, sequencing, and relevant constraints)
- Quality Requirements (reliability, maintainability, supportability, producibility, safety, extensibility, flexibility, reconfigurability, and interoperability)
- Human Factors (includes natural environment)
- Test (all types)
- Software Development Environment (tools, development approach, and personnel)
- Sustainment (support resources, software test equipment, and technical manuals)
In addition to identifying the above development areas, we estimated the level of engineering effort or emphasis that would be expended during the IDR and FDR. The estimation process was based on a small survey of experienced software personnel. Each person was asked to rate the amount of emphasis the specific development area should receive during IDR and FDR, using the scale "low," "medium," and "high." We then formed a general consensus to provide a relative idea of how much work should be expended in each area during IDR and FDR. These results can be used as a rule of thumb and as high-level guidance for planning purposes. For instance, one would expect a high level of emphasis for preliminary design during IDR and a low level of emphasis on preliminary design during the FDR. In fact, the work related to preliminary design during FDR would be "cleanup" in nature, such as clarifying any outstanding issues leftover from the IDR milestone.
Figure 3 shows the comparison of IDR and FDR engineering emphasis for each of the 12 areas we identified.
Figure 3: Engineering Emphasis for Different Development Areas
[Abstract] [1
Introduction] [2 IDR and FDR Definitions] [3
Engineering Emphasis] [4
IDR and FDR Criteria]
[5 Conclusion] [Appendix
A IDR Criteria] [Appendix
B FDR Criteria] [Appendix C Acronyms]
[References] [PDF
File]
4 IDR and FDR Criteria
The IDR and FDR are technically milestones, but the processes leading up to the milestones include a wide range of activities that must be accomplished before the actual milestone can be deemed a success. Therefore, the IDR and FDR milestones are capstone milestones for which a review is done on the engineering activities that preceded them. Thus, there must be criteria created for each milestone. In addition, preconditions (entry) and postconditions (exit) criteria for both IDR and FDR must be defined. The rationale for the criteria, the pre- and postconditions, and the actual criteria are described in this section.
4.1 Criteria Rationale
Many of today's large defense programs use Advanced Concept Technology Demonstrations (ACTDs) to gain understanding and evaluate the utility of a technology, develop a concept of operations for that technology, and expedite delivery of new capabilities to combat forces. ACTDs promote the rapid transition of the new technology into the appropriate phase of a formal acquisition program. Programs operating in an ACTD-like environment develop initial capabilities in prototype form. These prototypes, rather than merely acting as a proof of concept or refinement of requirements, are intended to provide lingering operational capability for extended operational evaluation and exploitation.
A system and software architecture and design must be developed in the TD phase of the acquisition life cycle and evaluated at IDR and FDR. The system representation must include a definition of the configuration items (CIs) for both hardware and software, how functionality is distributed across CIs, and how they interact to provide system-level capabilities.
These system-level capabilities include a number of necessary attributes that are architectural in nature. These attributes include but are not limited to security, flexibility, extensibility and reconfigurability, and interoperability. Evaluation of these architectural attributes must be supported by increasingly rigorous architectural reviews to verify that the desired emergent properties of the system will, indeed, be present when the system is fielded.
In addition to the emergent system properties, a number of workload, resource, and logistics attributes that are more characteristic of the design than the architecture need to be verified. This is because the TD phase is intended to provide operational utility that needs to be supported and evolved in the development phase. The architectural and design reviews can be combined for IDR and FDR.
Another important aspect of IDR and FDR criteria falls in the engineering and management area. As the program progresses, the comprehensive design and decision rationale should be recorded. Many times, this is not achieved. This is especially true when a program employs incremental or spiral-based development approaches. Often there are a collection of "hard" requirements that induce disproportionate risk on the system. It has been observed in both an incremental and spiral development of capabilities for demonstration, that there is a tendency to defer the "difficult requirements" to a later increment or spiral in order to produce partial capability for demonstration. Strong IDR and FDR criteria are needed to ensure that program development progresses in such a way that hard capabilities, which represent risks to the program, are associated with effective mitigation and development activities earlier in the project life cycle. In addition, these mitigation and development activities need to be well documented so that future program participants can track why and perhaps how certain decisions were made.
4.2 IDR and FDR Pre- and Postconditions
For both IDR and FDR, a set of conditions is defined for entry and exit into the IDR and FDR milestones. These criteria are demonstrable conditions in that they require something to be created, available, or accomplished before the IDR or FDR milestone can be achieved.
The proposed sets of IDR and FDR pre- and postconditions are found in Table 2 through Table 5. Each table contains two columns. The first column is labeled either "Preconditions" or "Postconditions." These are the lists of items that need to be created, available, or accomplished for IDR or FDR to either enter into the review or exit the review. The second column in each table is labeled "Potential UML Artifacts." This column provides some insight into common Unified Modeling Language (UML) artifacts that may be helpful in satisfying the IDR and FDR artifact requirements. Note that "Potential UML Artifacts" column may contain blank cells. This indicates that no applicable UML artifact exists to satisfy that condition. Although UML is the modeling and specification language of choice for many development efforts, it may be insufficient to document all aspects of your software development. In this case, other documentation such as specifications, interface control documents, and design documents should be used. In some instances, a "+Model analysis" entry is listed in the "Potential UML Artifacts" column. This indicates that an actual analysis of the model may provide additional information to meet the cited conditions.
Table 2: IDR Preconditions
Table 3: IDR Postconditions
Table 4: FDR Preconditions
4.3 IDR and FDR Criteria Summaries
In Section 3, Engineering Emphasis, we identified twelve areas of software development that form the basis of the IDR and FDR evaluation criteria. Note that not all twelve areas are represented in the pre- and postcondition criteria for IDR and FDR. For instance, "Detailed Design" is not discussed in Table 2 or Table 3, nor is "Preliminary Design" discussed in Table 4 or Table 5.
The first column in Table 6 summarizes the areas of concern addressed for IDR; the first column in Table 7 summarizes the areas of concern addressed in FDR. In each table, the second column lists potential UML artifacts. As with Table 2 through Table 5, blank cells in the column labeled "Potential UML Artifact" indicate that an applicable UML artifact for that area of concern does not exist.
Table 6: Summary IDR Software Areas of Concern
Table 7: Summary FDR Software Areas of Concern
4.4 CSCI Preliminary Design Criteria Example
Table 8 shows the basic criteria we identified for the CSCI Preliminary Design software development area, including the suggested detail/action and the potential UML artifacts for IDR. In this case, the area of concern includes functional flow, CSCI structure, interfaces, and COTS / GOTS / Reuse. A suggested detail is a top-level description of the functional flow for all software technical requirements document (TRD) requirements, including interfaces. Example UML artifacts are sequence or swim lane diagrams.
Table 8: Example IDR Software Evaluation Criteria
Due to the extensive nature of the IDR and FDR criteria, comprehensive lists of specific criteria are included in Appendices A and B, respectively.
4.5 Applying the Criteria
IDR and FDR are the culmination of the activities that generated the information required to exit each event. These formal, technical reviews are the result of significant work done at the working group level and in various intermediate reviews. depicts how the criteria can be applied at different levels of abstraction during a program. The bottom layer shows the reviews supported by the working group, integrated product team (IPT) or technical interchange meeting (TIM). The middle layer, or "wall walk," is the intermediate review at the CI or CSCI level, which is conducted in preparation for the formal IDR or FDR. The top of the pyramid represents the capstone IDR or FDR event.
Figure 4: IDR and FDR Validation Hierarchy
At the lowest level (the working group, IPT, and TIM level), peer reviews and artifact reviews are conducted by applying the suggested IDR or FDR criteria listed in the tables in Appendices A or B, respectively. All milestone preconditions must be met in order to start this phase. During this review, module or specific issues are raised, researched, resolved, and/or mitigated. After work is completed at this level, the reviews can proceed to the intermediate level or "wall walk" level.
The intermediate level is sometimes called "wall walk" because the data may be presented for review tacked to the walls of a large room. Reviewers walk around or along the wall and examine the material. The preconditions must be satisfied to enter this phase and the reviewers should ensure that the postconditions can be met by evaluating the data presented. This review is an overall, end-to-end review of major components. The major components usually comprise multiple components or modules reviewed at the working group level. Again, the suggested IDR or FDR criteria listed in the tables in Appendices A or B, respectively, is applied to the higher level components.
Finally, the capstone event, the IDR or FDR, is held. This is a high-level review of individual criteria that provides the evidence necessary for the milestone postconditions to be declared met. All the stakeholders are invited to the IDR and FDR. The Program Office has final say, with inputs from the stakeholders, as to whether the postconditions are met. These reviews vary in length, depending on the complexity of the program and how much of the program is covered in the review. In many instances, a large program is divided into segments with each segment holding its own IDR, FDR, and so on.
Abstract] [1
Introduction] [2 IDR and FDR Definitions] [3
Engineering Emphasis] [4
IDR and FDR Criteria]
[5 Conclusion] [Appendix A IDR Criteria]
[Appendix
B FDR Criteria] [Appendix C Acronyms]
[References] [PDF
File]
5
Conclusion
IDRs and FDRs held during the TD phase of a program can ensure that the required capability is demonstrated and that high risks have been mitigated. They provide a way to introduce rigor and formality into a program during the TD phase, ensuring a required level of progress, technical adequacy, and risk resolution on a technical, cost, and schedule basis has been achieved.
Readers of this technical note are encouraged to adapt the IDR and FDR criteria presented to their program's environment and needs. Users must also decide what non-UML artifacts are needed to round out the program's software documentation package. These adapted criteria can then be used to provide guidance to the contractors in the software area. In addition, the capstone milestones of IDR and FDR can be added to the Integrated Management Plan and Integrated Master Schedule of the program. This will ensure that the events are well known and that the activities necessary to achieve successful IDR and FDR are planned and accomplished.
[Abstract] [1
Introduction] [2 IDR and FDR Definitions] [3
Engineering Emphasis] [4
IDR and FDR Criteria]
[5 Conclusion] [Appendix A IDR Criteria]
[Appendix
B FDR Criteria] [Appendix C Acronyms]
[References] [PDF
File]
Appendix A IDR Criteria
IDR Software Evaluation Criteria
[Abstract] [1
Introduction] [2 IDR and FDR Definitions] [3
Engineering Emphasis] [4
IDR and FDR Criteria]
[5 Conclusion] [Appendix
A IDR Criteria] [Appendix
B FDR Criteria] [Appendix C Acronyms]
[References] [PDF
File]
Appendix
B FDR Criteria
FDR Software Evaluation Criteria
[Abstract] [1
Introduction] [2 IDR and FDR Definitions] [3
Engineering Emphasis] [4
IDR and FDR Criteria]
[5 Conclusion] [Appendix
A IDR Criteria] [Appendix
B FDR Criteria] [Appendix C Acronyms]
[References] [PDF
File]
Appendix C Acronyms
ACTD
Advanced Concept Technology Demonstration
CDR
Critical Design Review
CI
Configuration Item
COTS
Commercial Off-the-Shelf
CSCI
Computer Software Configuration Item
DoD
Department of Defense
DT&E
Developmental Test and Evaluation
ECP
Engineering Change Proposal
FDR
Final Design Review
GFP
Government Furnished Property
GOTS
Government Off-the-Shelf
HW
Hardware
IA
Information Assurance
IDR
Initial Design Review
IPT
Integrated Product Team
PDR
Preliminary Design Review
RMA
Reliability/Maintainability/Availability
SDD
System Development and Demonstration
SW
Software
TD
Technology Development
TIM
Technical Interchange Meeting
TRD
Technical Requirements Document
TRR
Test Readiness Review
USAF
United States Air Force
[Abstract] [1
Introduction] [2 IDR and FDR Definitions] [3
Engineering Emphasis] [4
IDR and FDR Criteria]
[5 Conclusion] [Appendix
A IDR Criteria] [Appendix
B FDR Criteria] [Appendix C Acronyms]
[References] [PDF
File]
References
| Department of Defense. MIL-STD-1521B (USAF) Military Standard Technical Reviews and Audits for Systems, Equipments, and Computer Software. (Cancelled April 10, 1995) (1985). |
|
| Department of Defense. MIL-STD-973 Military Standard Configuration Management. (Cancelled September 30, 2000) (1992). |
|
| Department of Defense. DoD Instruction Operation of the Defense Acquisition System (DoDI 5000.2). (2003). |
|
| U.S. Air Force. Software Management Guidebook, Version 0.9. (2004). |
|
 |
[Abstract] [1
Introduction] [2 IDR and FDR Definitions] [3
Engineering Emphasis] [4
IDR and FDR Criteria]
[5 Conclusion] [Appendix
A IDR Criteria] [Appendix
B FDR Criteria] [Appendix C Acronyms]
[References] [PDF
File]
This work is sponsored by the U.S. Department of Defense.
The Software Engineering Institute is a federally funded research and development center sponsored by the U.S. Department of Defense.
Copyright 2006 Carnegie Mellon University.
NO WARRANTY
THIS CARNEGIE MELLON UNIVERSITY AND SOFTWARE ENGINEERING INSTITUTE MATERIAL IS FURNISHED ON AN "AS-IS" BASIS. CARNEGIE MELLON UNIVERSITY MAKES NO WARRANTIES OF ANY KIND, EITHER EXPRESSED OR IMPLIED, AS TO ANY MATTER INCLUDING, BUT NOT LIMITED TO, WARRANTY OF FITNESS FOR PURPOSE OR MERCHANTABILITY, EXCLUSIVITY, OR RESULTS OBTAINED FROM USE OF THE MATERIAL. CARNEGIE MELLON UNIVERSITY DOES NOT MAKE ANY WARRANTY OF ANY KIND WITH RESPECT TO FREEDOM FROM PATENT, TRADEMARK, OR COPYRIGHT INFRINGEMENT.
Use of any trademarks in this report is not intended in any way to infringe on the rights of the trademark holder.
Internal use. Permission to reproduce this document and to prepare derivative works from this document for internal use is granted, provided the copyright and "No Warranty" statements are included with all reproductions and derivative works.
External use. Requests for permission to reproduce this document or prepare derivative works of this document for external and commercial use should be addressed to the SEI Licensing Agent.
This work was created in the performance of Federal Government Contract Number FA8721-05-C-0003 with Carnegie Mellon University for the operation of the Software Engineering Institute, a federally funded research and development center. The Government of the United States has a royalty-free government-purpose license to use, duplicate, or disclose the work, in whole or in part and in any manner, and to have or permit others to do so, for government purposes pursuant to the copyright license under the clause at 252.227-7013.
For information about purchasing paper copies of SEI reports, please visit the publications portion of our Web site (http://www.sei.cmu.edu/publications/pubweb.html).
[Abstract] [1
Introduction] [2 IDR and FDR Definitions] [3
Engineering Emphasis] [4
IDR and FDR Criteria]
[5 Conclusion] [Appendix
A IDR Criteria] [Appendix
B FDR Criteria] [Appendix C Acronyms]
[References] [PDF
File]