In the development and fielding of software-reliant systems, testing plays a critical role.  In spite of the best efforts of requirements engineers, designers, and programmers, faults are inevitably introduced that are only discovered and weeded out by running the system and comparing what it does to what it is supposed to do.   Depending on the criticality of the system, testing can occupy 40%, 70%, even 90% of a project’s schedule and budget.

Currently, architecture plays a minor role in most testing processes.  Running code is tested against requirements.  We believe architecture has a more important role to play in testing, and our research is aimed at fleshing out that role and measuring the benefits of using architecture to produce better testing outcomes.  A better testing outcome is one in which the efficacy of testing is increased for the same cost, or the same efficacy is achieved at a decreased cost.

Our primary research work for FY11 involves defining, creating, and demonstrating the use of testability profiles for architectural design approaches. 

An architectural design approach (ADA) is a combination of architectural styles, patterns, and tactics chosen by the architect to employ in the solution design for a system.

A testability profile for an ADA consists of four parts:

1.       A set of observables associated with using the ADA.  These are properties of an implemented system that can be used to verify that the claimed ADA was in fact employed in the design of a system.  For example, if a service-oriented architecture style is used, observables would include the set of identifiable services as well as the componentry used to facilitate their interaction and communication.

2.       A fault model for the ADA.  A fault model is a list of faults that are associated with a design approach.  The fault model has two parts. 

o    First, it lists faults that can occur by dint of choosing the ADA.  For example, in a pipe-and-filter system, pipes must not change the order or value of the data in the data streams that they transmit, and they must not communicate with other pipes in any way.  If they do, this is a fault (and moreover, it is a fault unique to the pipe-and-filter style).

o    Second, the fault model lists faults that cannot occur by dint of the chosen ADA.  For example, if the architect has chosen a state machine as the design approach for encapsulating the control logic of a module, a subsystem, or the entire system, then no control logic errors are possible outside of the state machine’s encapsulating component.

3.       Available analysis corresponding to the fault model.  This part of the profile details any available tools and methods with which to draw conclusions about systems that are compliant with that architecture.  The analysis may be architecture based – that is, take as input a model of the architecture or a relevant part of it – or it may be code based.

4.       Tests made redundant based on the analysis.  If the analysis described above is done for an architecture that includes the ADA, then it should make certain tests for the corresponding faults unnecessary, or allow them to be de-emphasized in the testing procedures.  For example, if analysis can show that, in any architecture-compliant implementation, deadlock is impossible then it should be unnecessary to run tests to detect deadlock.