Risk/Misfit Redux
ROBERT C. SEACORD

Introduction

In our book, Building Systems from Commercial Components [1], we introduced Risk/Misfit as a decision aid for exposing and quantifying the cost and risks of using and repairing components. Risk/Misfit requires that the use of a particular component feature be justified according to the design risk that arises from not using the feature.

There are a number of uses for Risk/Misfit. In the book, we provide an extended illustration for selecting among strategies for repairing a design risk caused by the lack of a component feature. Another use is a best-fit evaluation of a component ensemble.

Component ensembles are a central theme of our approach to building systems from commercial components (they were covered in both my second and third quarter columns last year). Component ensembles are collections of compatible components that satisfy a design problem. Model problems (the subject of my second quarter 2001 column) can be used to establish feasibility, that is, that a component ensemble can satisfy a design problem within a given set of constraints. However, when multiple design ensembles are feasible, we must use Risk/Misfit or another comparable technique to select an optimal ensemble.

Ensemble Evaluation

To select an optimal component ensemble with Risk/Misfit, it is first necessary to establish common criteria against which the ensembles can be evaluated. It is insufficient to simply list the features provided by each ensemble and quantify the costs and risks of not having these features, as the feature sets are driven by the ensembles and cannot provide a common basis for evaluation. Instead, we invert the Risk/Misfit technique we applied to the selection of repair strategies, and start by listing the major risks inherent in the design problem that the ensemble is meant to solve. As in the model problem process, these risks can be iteratively extended during the evaluation process, as long as common criteria are reapplied to each ensemble under consideration.

In this modified process, we then calculate the maximum risk for each ensemble. In Table 1, we list three project risks and assign values to each of these by defining a scale (in this case, 1–100) and selecting a value for each risk. In the third column, we estimate the value of eliminating each risk. This value is obtained by asking yourself (or your manager), “How much would we pay to completely eliminate risk X?” If this is not a positive number, the risk is probably not significant enough to be considered in the evaluation.

Table 1 :  Calculating Ensemble Risk

Project Risks	MR	Worth ($) to Eliminate	V ($/R)	Feature Set	RR	r(r)
Inadequate performance	92	$956,000	$10,391	E1: F2, F4	5	$51,955
				E2: F1	12	$124,692
				E3: F3, F7	6	$62,346
Lack of software engineers qualified to work with proposed technologies	87	$270,000	$3,103	E1: F3	10	$31,030
				E2: F4, F5	0	$0
				E3: F1, F2	3	$9,309
Inadequate data confidentiality	79	$456,000	$5,772	E1: F6	10	$57,720
				E2: F8	22	$126,984
				E3: —	79	$456,000

The fourth column in our table simply involves a small amount of math. In it, we divide the total worth to eliminate the risk by the maximum risk (MR), resulting in a value per risk unit.

For example, Table 1 lists “inadequate performance” as a project risk. This risk is assigned a maximum risk value of 92, and it would be worth a whopping $956,000 for this problem to simply go away. These two figures are used to calculate the worth of eliminating each risk unit at $10,391. 

Now that the preliminaries are out of the way, we can get to the fun part. We now identify those features of each ensemble that address each project risk. These features must be apparent in the application of the component ensemble to the design problem. Many component features are apparent in the overall ensemble, while others are not. Only those component features that are apparent in the solution should be used in the Risk/Misfit evaluation. For example, if your product supports a PKI feature, but you do not intend to use it in the ensemble, do not evaluate that feature.

Some ensembles might demonstrate emergent properties that result from combining products. If an emergent property produces a positive effect, it should be considered as a feature of the ensemble.

In the next column, we calculate the residual risk (RR) remaining once the availability of these ensemble features is considered. For example, in our first ensemble (E1), features F2 and F3 both address the risk of inadequate performance. However, these solutions are not perfect (solutions seldom are) and some residual risks remain. These residual risks are captured in the RR column. The final column, labeled “r(r),” converts residual risk to dollars. In this case, the function ρ(r) multiplies the residual risk number by the cost per unit risk value we calculated for column 4.

Concluding the Evaluation

Once we have completed this analysis for each project risk, we can calculate the total residual risk for each ensemble by simply summing the r(r) values for each ensemble for each risk. For example, the total residual risk for ensemble E1 would be equal to Sr(r): $51,955 + $31,030 + $57,720, or $140,705. Total residual risk costs for each ensemble are shown in the third column of Table 2.

At this point, we could compare each ensemble and select the one with the lowest residual risk. However, we would also like to include another factor, which is the total cost of ownership (TCO) for the ensemble. The TCO can include the cost of licensing all the products, integration costs, administration costs, training costs, and other costs related to buying, learning, and using the product. How you calculate the TCO is largely up to you and your organization, since organization and other factors can affect which costs may or may not be of concern to you in your decision-making process.

Once we have the TCO for each ensemble, we can add these costs to the overall residual-risks costs for each ensemble. Again, the ensemble with the lowest overall cost and risk would normally be selected. In Table 2, Ensemble E2 would be selected, as it has the lowest combined TCO and residual risk.

Table 2 : Ensemble Total Cost of Ownership

Ensemble	TCO	Sr(r)	TCO + Sr(r)
E1	$350,000	$140,705	$490,705
E2	$247,000	$151,676	$370,000
E3	$423,000	$527,655	$950,655

Summary

While the above example is somewhat simplistic, it should provide the general sense of how Risk/Misfit can be applied to the best-fit evaluation of component ensembles. The area in which this example could be further expanded is in the evaluation and selection of repair strategies. For example, it may be that the residual risk for many of the ensembles described above are still quite high. It is quite possible that repairs can be made to further reduce these residual risks. Also, repairs could be made to reduce risk resulting from missing features (that is, the residual risk is the same as the maximum risk). The cost of making these repairs and the residual risk remaining after their completion must also be accounted for in the evaluation. 

References

[1]	Wallnau, Kurt C.; Hissam, Scott A.; and Seacord, Robert C. Building Systems from Commercial Components. Boston: Addison-Wesley, 2002, ISBN: 0201700646.

About the Author

Robert C. Seacord is a senior member of the technical staff at the SEI and an eclectic technologist. He is coauthor of the book Building Systems from Commercial Components as well as more than 40 papers on component-based software engineering, Web-based system design, legacy system modernization, component repositories and search engines, security, and user interface design and development.

The views expressed in this article are the author's only and do not represent directly or imply any official position or view of the Software Engineering Institute or Carnegie Mellon University. This article is intended to stimulate further discussion about this topic.