It is difficult to assure the safety, security, or reliability of net-centric systems of systems because of their size, complexity, and continuing evolution—and because they can exhibit undesired and unanticipated emergent behavior (actions of a system as a whole that are not simple combinations of the actions of the individual constituents of the system.)
Traditional software and systems engineering techniques, including conventional test and evaluation approaches, cannot provide the justified confidence needed. The SEI is developing an assurance case methodology to augment testing and evaluation.
The assurance case provides a means to structure the reasoning that engineers use implicitly to gain confidence that systems will work as expected. It also becomes a key element in the documentation of the system and provides a map to more detailed information.
The concept of an assurance case has been derived from the safety case, a construct that has been used successfully in Europe for over a decade to document safety for nuclear power plants, transportation systems, automotive systems, and avionics systems.
The activities required to construct an assurance case are largely those that a conscientious developer of mission-critical systems would normally undertake. But the assurance case, and the assurance plan, highlights factors relating to system dependability in a reviewable form.
Much like a legal case presented in a courtroom, an assurance case requires arguments linking evidence with claims of conformance to dependability-related requirements.
- A claim embodies what we want to show.
- An argument tells us why we believe a claim has been met.
- Evidence supporting the argument that may take many forms—including test results, formal analyses, simulation results, fault-tree analyses, hazard analyses, modeling, and inspections.
It is important that an assurance case be reviewable, which means that having a single claim (e.g., "The system does what it's supposed to do") and a single complex argument that links myriad evidence to the claim are not appropriate. Instead of taking such a giant step, the claim is typically broken into subclaims, each of which can potentially be broken into yet another level of subclaims until the step to the actual evidence that supports that subclaim is almost obvious. This structure is shown in Figure 1.
Figure 1: A Structured Argument
Graphical Representation of the Structured Argument
The structured argument shows how claims are broken down into subclaims and supported by evidence. Figure 2 shows a fragment of a possible assurance case for a system. This notional example starts with a general claim about the project's dependability and will be developed to incorporate the timing requirement as stated in the hypothetical systems requirements documents. For purposes of exposition, the example uses a graphical notation called Goal Structuring Notation (GSN), but as long as the structure of the argument is clear, any method will do. We are using a notation that was originally developed by T. P. Kelly (University of York) for use in documenting safety cases.
Figure 2: An Assurance Case Fragment
Starting at the top of the diagram, the claim, "Dependable System," expresses what the rest of the case is intended to support. Claims are stated as simple predicates—that is, as statements that can be true or false. (In this notation, claims are expressed in rectangular boxes.) The claim makes a reference to satisfying Project X's "dependability requirements." The case documents where these requirements can be found in a context element, which is expressed in a box with rounded corners.
The context elements are the means by which the case references detailed artifacts that have been developed elsewhere. In this instance, the context element references the document X and derived dependability requirements that are stated in document Y.
In the next step, the assurance case documents the argumentation strategy that is going to be used—namely, the case will address each of the dependability attributes separately. (Strategy elements are written in parallelograms.) The first step in implementing this strategy is to articulate the claims for each attribute (these are the claims below the strategy element). These elements have a diamond at the bottom to indicate that the argument requires further development (i.e., the notation indicates when the argument is complete and when further work is needed). This notation allows assurance cases to be developed incrementally.
The strategy element is not part of the argument; it is present so that reviewers can more easily understand the nature of the argument. There are other types of notations that are useful in helping to understand an argument. The context element is one; there are also elements for documenting assumptions underlying the argument or giving an explanation (a "justification") for making a particular claim, when it might not be obvious otherwise. This is not a complete discussion of developing an assurance case using GSN. GSN-represented assurance cases can be built in most any graphical shape rendering system.
In Figure 3, we have chosen to further develop the reliability claim, and the selected strategy is to address the reliability of each major subsystem in the project. This argument requires not only reliability claims for each subsystem, but claims that all relevant subsystems have been addressed and that there are no interactions between the subsystems that would affect the overall system reliability.
Figure 3: Subsystem Reliability
Creating a Structured Argument
Creating a structured argument is best accomplished in a top-down manner, as illustrated so far by our example.
The process starts by identifying top-level claims to be made. These top-level claims should provide the "take home" message for the reviewers and should (if supported) convince them that the required attributes have been satisfied. The top-level claims should be statements that do not consider implementation details. "System X meets its reliability requirements" is a good top-level claim while "System X meets its reliability requirements through the use of a hot standby" is not.
The implementation details are better treated in the argument supporting the claim. That said, it is also important that the top-level claim not be oversimplified. "System X is reliable" is not as useful a top-level claim as "System X meets its reliability requirements." Getting top-level claims right is important. They form the seed from which the arguments develop and if they don't contain the right concepts or include specific details the argument's usefulness may be impaired. A claim is properly worded if it is (ultimately) either true or false.
|Properly Worded Claims
||Poorly Worded Claims (and why they are not well formed)|
|Component X has no critical failure rates
||Hazard Log for System Y (describes an entity)|
|All identified hazards for System Y have been sufficiently mitigated
||Perform Fault Tree Analysis of Subsystem S1 (an action, not a statement)|
The text that describes a claim must be reasonably succinct if the argument is to be reviewable. Yet it is often necessary to provide additional information that is not directly part of the argument. For instance, the claim may contain terms that are not generally known, or it may refer to a standard or a requirements document. Context is used to provide this additional information. For example: "Context: The system reliability definition being applied can be found in SAE Standard XX, page YY." The context should never be a part of the actual argument. Justification and assumption cues are used similarly—to provide additional information to make the argument more understandable.
Identifying Strategy. As defined above, the strategy is an additional cue that helps the reader understand the form that an argument is going to take. Instead of being true or false statements, as the claims and subclaims are, the strategy provides information on how to substantiate the stated claim. The strategy can take many forms, but it is often simply a matter of presentation.
The example in Figure 3 shows a strategy that might be termed "divide and conquer" in that it is breaking the problem down into several smaller problems, the total of which cover all possibilities— and each of which is likely to be easier to deal with than the argument as a whole.
The strategy can be explicit (as it is in Figure 3) or implicit (as it would be if we had not added the strategy in that figure). Sometimes the strategy is obvious from the layout of the subclaims, in which case an implicit strategy is fine. However, if the relationship between a claim and a set of subclaims has any complexity or if the strategy being used requires additional context, it should be explicit.
Strategies should not contain claims. They should be phrased with respect to the argument, not with respect to the design, testing, or analysis approach. Thus, it would be wrong to say "Use Byzantine Agreement protocol." Rather the strategy should be "Argument by appeal to the Byzantine Agreement protocol." It should be possible to remove a strategy cue and not affect the argument being made.
As with claims, it is perfectly reasonable and, in fact, often helpful to associate context, justification, or assumptions with the strategy.
Elaborating the Strategy. A strategy is elaborated by providing a series of subclaims that fulfill the selected approach. For a strategy ranging over all subsystems, for instance, claims must be made that cover each individual subsystem. For a quantitative results strategy, there must be quantitative claims.
Elaborating these claims is exactly the same as elaborating the top-level claim. If a good strategy has been chosen and the basis for the strategy is clear, this can be very straightforward.
Evidence. Eventually, as the expansion continues, there will come a point where a claim needs no further refinement or explanation. In such cases, it is sufficient to refer to the evidence that supports that claim. An example is given later. One caveat: what may seem obvious (and therefore not requiring refinement) to the creator of an assurance case may not be at all obvious to the typical reader of the case. When in doubt it is best to err on the side of providing too many steps rather than too few.
Textual Representation. While perhaps not as easy to review as a graphical representation, a textual representation of an assurance case can be created, using indentation and font changes to indicate structure. The main advantage of using text is that it requires little or no investment in tools. The documents can be prepared in word processing or spreadsheet software. The key things to remember are that it is important to use simple language and short sentences and structure the text much like an extended outline. Whichever form is used, graphical or textual, the resulting artifact may be thought of as a map for the overall argument structure. Where appropriate, each node in the case will have a link to external supporting documentation that thereby becomes part of the case.
Figure 4 shows the assurance case fragment presented in Figure 2 using a text-based representation.
Figure 4: A Text Version of the Assurance Case Fragment
Tools and Notation. Although we show the development of an assurance case using the GSN both graphically and textually, there are other available notations (such as Adelard Safety Case Development). As long as the structure of the argument is clearly presented, the method of exposition of the case and the tools used to develop the case are at the discretion of the project.