Assurance Case and Plan Preparation
Creating a Structured Argument
The Evolution of an Assurance Plan and Case
Example of an Evolving Assurance Case
Introduction
The activities required to construct an assurance case are largely those that a conscientious developer of mission-critical systems would undertake in the normal course of business. But the assurance plan, and more significantly the assurance case, highlights factors relating to system dependability in a reviewable form.
Much like a legal case presented in a courtroom, an assurance case requires arguments linking evidence with claims of conformance to dependability-related requirements. The evidence may consist of test results, formal analyses, simulation results, fault-tree analyses, hazard analyses, modeling, and inspections. The argument is the explanation of how the available evidence can reasonably be interpreted as indicating the required levels of dependability. Arguing that an artifact is dependable without evidence is unfounded. Evidence without supporting arguments is unexplained.
Creating and presenting assurance cases in a form in which they can be reviewed by outside observers requires some care. While the case can be presented textually, a graphical representation is often much more easily understood by reviewers and more useful to the developers and maintainers of the artifact under review.
return to top
Argument Structure
This section presents a method for graphically representing and developing an assurance case; the method has been used in Europe for over a decade to document safety cases for nuclear power plants, transportation systems, automotive systems, and avionics systems. [1]
Assurance Case Elements
An assurance case consists of claims, arguments, and evidence. A claim embodies what we want to show; an argument tells us why we believe a claim has been met, based upon evidence that may take many forms including test results, simulation results, or analysis. It is important that an assurance case be reviewable, which means that having a single claim (e.g., "The system does what it's supposed to do") and a single complex argument that links myriad evidence to the claim are not appropriate. Instead of taking such a giant step, the claim is typically broken into subclaims, each of which can potentially be broken into yet another level of subclaims until the step to the actual evidence that supports that subclaim is almost obvious. This structure is shown in Figure 1.
Figure 1: A Structured Argument
Graphical Representation of the Structured Argument
The structured argument shows how claims are broken down into subclaims and supported by evidence. Figure 2 shows a fragment of a possible assurance case for a system. This notional example starts with a general claim about the project's dependability and will be developed to incorporate the timing requirement as stated in the hypothetical systems requirements documents. For purposes of exposition, the example uses a graphical notation called Goal Structuring Notation (GSN), but as long as the structure of the argument is clear, any method will do. (Also see Textual Represenation .)We are using a notation that was originally developed by T. P. Kelly (University of York) for use in documenting safety cases.
Figure 2: An Assurance Case Fragment
Starting at the top of the diagram, the claim, "Dependable System," expresses what the rest of the case is intended to support. Claims are stated as simple predicates (i.e., statements that can be true or false). (In this notation, claims are expressed in rectangular boxes.) The claim makes a reference to satisfying Project X's "dependability requirements." The case documents where these requirements can be found in a context element, which is expressed in a box with rounded corners. The context elements are the means by which the case references detailed artifacts that have been developed elsewhere. In this instance, the context element references the document X and derived dependability requirements that are stated in document Y.
In the next step, the assurance case documents the argumentation strategy that is going to be used -- namely, the case will address each of the dependability attributes separately. (Strategy elements are written in parallelograms.) The first step in implementing this strategy is to articulate the claims for each attribute (these are the claims below the strategy element). These elements have a diamond at the bottom to indicate that the argument requires further development (i.e., the notation indicates when the argument is complete and when further work is needed). This notation allows assurance cases to be developed incrementally.
The strategy element is not part of the argument; it is present so that reviewers can more easily understand the nature of the argument. There are other types of notations that are useful in helping to understand an argument. The context element is one; there are also elements for documenting assumptions underlying the argument or giving an explanation (a "justification") for making a particular claim, when it might not be obvious otherwise.
This is not a complete discussion of developing an assurance case using GSN. GSN-represented assurance cases can be built in most any graphical shape rendering system.
In Figure 3, we have chosen to further develop the reliability claim, and the selected strategy is to address the reliability of each major subsystem in the project. This argument requires not only reliability claims for each subsystem, but claims that all relevant subsystems have been addressed and that there are no interactions between the subsystems that would affect the overall system reliability. We will expand the claim about system Y's reliability later.
Figure 3: Subsystem Reliability
Before continuing further with this example, we will discuss the general process of creating an assurance case using this approach.
return to top
Creating a Structured Argument
Claims
Creating a structured argument is a relatively straightforward process, best accomplished in a top-down manner, as illustrated so far by our example. The process starts by identifying top-level claims to be made. These top-level claims should provide the "take home" message for the reviewers and should (if supported) convince them that the required attributes have been satisfied. The top-level claims should be straightforward statements that do not consider implementation details. "System X meets its reliability requirements" is a good top-level claim while "System X meets its reliability requirements through the use of a hot standby" is not.
The implementation details are better treated in the argument supporting the claim. That said, it is also important that the top-level claim not be oversimplified. "System X is reliable" is not as useful a top-level claim as "System X meets its reliability requirements." Getting top-level claims right is important. They form the seed from which the arguments develop and if they don't contain the right concepts or include specific details the argument's usefulness may be impaired.
The following are examples of claims that are properly worded:
- Component X has no critical failure rates
- All identified hazards for System Y have been sufficiently mitigated
The following are examples of claims that are poorly worded:
- Hazard Log for System Y (describes an entity)
- Perform Fault Tree Analysis of Subsystem S1 (an action, not a statement)
A claim is properly worded if it is (ultimately) either true or false.
Establishing Context
The text that describes a claim must be reasonably succinct if the argument is to be reviewable. Yet it is often necessary to provide additional information that is not directly part of the argument. For instance, the claim may contain terms that are not generally known, or it may refer to a standard or a requirements document. Context is used to provide this additional information. For example: "Context: The system reliability definition being applied can be found in SAE Standard XX, page YY." The context should never be a part of the actual argument. Justification and assumption cues are used similarly—to provide additional information to make the argument more understandable.
Identifying Strategy
As defined above, the strategy is an additional cue that helps the reader understand the form that an argument is going to take. Instead of being true or false statements, as the claims and subclaims are, the strategy provides information on how to substantiate the stated claim. The strategy can take many forms, but it is often simply a matter of presentation. The example in Figure 3 shows a strategy that might be termed "divide and conquer" in that it is breaking the problem down into several smaller problems, the total of which cover all possibilities— and each of which is likely to be easier to deal with than the argument as a whole.
The strategy can be explicit (as it is in Figure 3) or implicit (as it would be if we had not added the strategy in that figure). Sometimes the strategy is obvious from the layout of the subclaims, in which case an implicit strategy is fine. However, if the relationship between a claim and a set of subclaims has any complexity or if the strategy being used requires additional context, it should be explicit.
Strategies should not contain claims. They should be phrased with respect to the argument, not with respect to the design, testing, or analysis approach. Thus, it would be wrong to say "Use Byzantine Agreement protocol." Rather the strategy should be "Argument by appeal to the Byzantine Agreement protocol." It should be possible to remove a strategy cue and not affect the argument being made.
As with claims, it is perfectly reasonable and, in fact, often helpful to associate context, justification, or assumptions with the strategy.
Elaborating the Strategy
A strategy is elaborated by providing a series of subclaims that fulfill the selected approach. For a strategy ranging over all subsystems, for instance, claims must be made that cover each individual subsystem. For a quantitative results strategy, there must be quantitative claims.
Elaborating these claims is exactly the same as elaborating the top-level claim. If a good strategy has been chosen and the basis for the strategy is clear, this can be very straightforward.
Evidence
Eventually, as the expansion continues, there will come a point where a claim needs no further refinement or explanation. In such cases, it is sufficient to refer to the evidence that supports that claim. An example is given later. One caveat: what may seem obvious (and therefore not requiring refinement) to the creator of an assurance case may not be at all obvious to the typical reader of the case. When in doubt it is best to err on the side of providing too many steps rather than too few.
Textual Representation
While perhaps not as easy to review as a graphical representation, a textual representation of an assurance case can be created, using indentation and font changes to indicate structure. The main advantage of using text is that it requires little or no investment in tools. The documents can be prepared in word processing or spreadsheet software. The key things to remember are that it is important to use simple language and short sentences and structure the text much like an extended outline. Whichever form is used, graphical or textual, the resulting artifact may be thought of as a map for the overall argument structure. Where appropriate, each node in the case will have a link to external supporting documentation that thereby becomes part of the case.
Figure 4 shows the assurance case fragment presented in Figure 2 using a text-based representation.
Figure 4: A Text Version of the Assurance Case Fragment
Tools and Notation
Although we show the development of an assurance case using the GSN both graphically and textually, there are other available notations (such as Adelard Safety Case Development[2]). As long as the structure of the argument is clearly presented, the method of exposition of the case and the tools used to develop the case are at the discretion of the project.
return to top
The Evolution of an Assurance Plan and Case
Relationship of Assurance Plan to Assurance Case
The assurance plan for a project defines the kinds of software assurance activities that will be accomplished at different phases of a project. These activities focus on the dependability attributes of a system, not its functional attributes, although deficiencies in functional attributes may affect system qualities such as reliability, availability, and safety. By and large, the dependability attributes of interest are attributes of the system as a whole (e,g., reliability or safety). These attributes depend on the interactions among system components. For assurance purposes, it is generally not sufficient to show that system components work correctly individually or in subsystems; it is also necessary to address component interactions to ensure a system's overall dependability attributes.
The assurance plan describes assurance case preparation activities undertaken to ensure that:
- Requirements for dependability are understood, analyzed for conflicts that may require tradeoffs, and implemented by the development organization.
- Resources are allocated to production of the assurance case.
- Task assignments, skills, budget, and schedule are allocated appropriately.
- The assurance case remains accurate and current.
- The assurance case is presented effectively.
The next sections present examples of the information that should appear in an assurance plan and an assurance case at different phases of a project's life cycle. The phases considered are Requirements, Design, Integration and Evaluation, and Deployment. The use of these terms does not imply a waterfall development model: the intent is to characterize how much is known about the system as its development evolves and what the implications would be for an assurance plan and case.
Requirements Phase
In the requirements phase, a description of the dependability attributes relevant to the project should be created. By the end of this phase, the plan will document (or reference) defined dependability requirements, and,in a general fashion, what activities will be undertaken in subsequent phases to demonstrate that the requirements will be or have been met. For example, the plan should focus on showing how:
- The project's dependability requirements can be demonstrated within the available time and schedule.
- Requirements (including derived requirements) will be evaluated for consistency and completeness, especially for those requirements or areas likely to affect mission success or safety.
- The documents to be generated in this or later phases will support the assurance case for the project. For example, software functions whose failure would critically impact system dependability will need to receive particular attention in the fully developed assurance case. The plan should describe the approach that will be taken to identify these critical functions.
The assurance plan should also discuss staffing and scheduling issues related to achieving the dependability requirements and maintaining the plan/case.
Design Phase
As a design evolves, it will be possible to create a more meaningful assurance plan and case. For example, the assurance case (at the design stage) will typically address how the software has been structured to cope with various hazards introduced by the environment and possible software component failures. The plan should discuss the extent of the planned hazard analyses or software failure modes and effects analysis (FMEAs). The case will then reference these analyses to show how the hazards and potential failure modes have been mitigated by the design. If special analyses and simulations are to be undertaken to verify various parts of the system design, the nature of these should be described in the plan. The results of these analyses will then be referenced in the case as evidence supporting particular dependability claims.
Toward the end of the design phase, it should be possible to provide a fairly complete set of dependability claims that will be supported by evidence gathered during the implementation and integration/test phases. By developing such an assurance case structure at this point, it will be clearer what effort is needed later to demonstrate the validity of the claims and how this effort will be effective in providing such support. For example, the strategy to be used for generating test cases is affected by how the test cases support the dependability claims. The strategy should be documented in the plan; the implementation of the testing strategy and the actual test case results will later be incorporated into the case.
Integration and Evaluation Phase
After integration and evaluation, the assurance case will show the detailed arguments that relate evidence gathered during this phase to the claims of assurance throughout the system. Typically the case will be augmented with test results.
Deployment Phase
For the assurance case to be useful, it must be maintained over time. The assurance plan for the deployment phase should discuss the processes and procedures to be used for keeping the assurance case current in the face of a changing system, possibly with multiple configurations. Examining the current case can help determine if proposed modifications will invalidate or change arguments and claims and, if so, which parts of the case will need to be updated. Further, if parts of the system prove to be undependable even in the face of a well-developed case, it will be important to analyze the case to understand which chain of evidence-argument-claim proved invalid.
return to top
Example of an Evolving Assurance Case
Expansion of a Reliability Claim
The expansion of the claim that "The Y subsystem satisfies its reliability requirements" could proceed as shown in Figure 5.
Figure 5: A Further Expansion of the Fragment Introduced in Figure 2
Here we have chosen to show that subsystem Y is reliable by addressing possible hazards that might impair Y's reliability. The claims at the bottom of Figure 5 cover only a few of the hazards that might be listed. They might be suitable for the requirements phase in which details of the design are not available. The hazard analysis document at this phase might just be an extract from other requirements documents and as this document is revised, the assurance case will have to be updated as well.
Expansion Suitable for the Design Phase
For purposes of this example, we will expand the claim, "Y Timing Budgets are Satisfied." The expansion is shown in Figure 6.
Figure 6: Expansion Suitable for the Pre-Design Phase
The strategy is to appeal to analytical and testing results. The context element for the "Y Schedulability Analysis" claim indicates the type of analysis intended. The case at this stage is sufficiently developed considering that the actual or detailed design of the software has not yet been completed. The assurance plan at this stage would discuss the kinds of analyses being contemplated and who will produce the documents referenced in the context elements.
Toward the end of the design phase, the case might look something like that shown in Figure 7.
Figure 7: Expansion After Design
This expands the context element for "Y Schedulability Analysis" to reference a document giving the detailed information required to conduct a schedulability analysis. The diagram also includes the results of the schedulability analyses, in "Ev: Y Analysis Results." (The circle is the shape used to describe evidence supporting claims.) The claim, "Y Testing," has been expanded to show that test results will be used for two purposes: to show that the schedulability model and analyses are consistent with the test results (this strengthens the validity of the schedulability analyses) and to show that test results indicate no violations of timing budgets. We have also added a justification element (in the oval shape) indicating why the selected test cases will be useful in demonstrating that timing constraints are not violated. An explanation of when and how this justification will be created should appear in the updated assurance plan associated with this case.
Expansion for Integration and Evaluation Phase
Finally, toward the end of the test and evaluation phase, the assurance case would look something like Figure 8. The primary differences between this view and Figure 7 are the addition of evidence elements for the previously unexpanded claims and the adjustment of context and justification elements to reference the current versions of materials.
It is important to note that all developers of such a system, even in the absence of a requirement to present an assurance case, would already be doing extensive testing aimed at showing that the timing constraints were met. Most developers would also already be doing a formal schedulability analysis and would correlate the results as shown here. One difference is that the assurance case provides a structure for the already developed evidence that allows an outsider to more efficiently review the work product and makes it easier to convince them that the requirements have been met. The amount of time spent in creating the diagrams summarizing the assurance case is small compared to the amount of time spent in generating the evidence and the detailed analyses serving as context or justification for claims. The effort to generate evidence is traditionally expended anyway; the assurance case simply provides a structure that helps integrate and clarify why the various forms of evidence support the claims.
Figure 8: Expansion After Implementation
[1] Kelly, T.P. & Weaver, R.A. "The Goal Structuring Notation: A Safety Argument Notation." Proceedings of the Dependable Systems and Networks 2004 Workshop on Assurance Cases. Florence, Italy, July 2004 http://www-users.cs.york.ac.uk/~rob/papers/DSN04.pdf
[2] Adelard. The Adelard Safety Case Development Manual. http://www.adelard.com/web/hnav/resources/ascad
Charles B. Weinstock
This page was last modified on June 22, 2007