Evaluating Risks in COTS Acquisition Programs

NEWS AT SEI

This article was originally published in News at SEI on: June 1, 2000

The importance of managing risk is well understood in the software engineering community. Department of Defense (DoD) directives and mandates, such as DOD 5000.1 and 5000.2m, specify the use of risk reduction activities. And the SEI's Software Risk Evaluation (SRE) has been a significant part of acquisition for several years.

In an acquisition that will include extensive use of commercial off-the-shelf (COTS) products, several problems emerge that are not present in non-COTS-intensive acquisitions. For example, the requirements process must become more flexible, yielding to the realities of commercial products, such as the inability to control when products are released, their features, and their ability to interface with other products. Such problems contribute to a program manager's loss of control, and hence, added risk. To help managers manage these risks, the SEI has developed a COTS Usage Risk Evaluation (CURE).

About CURE

CURE is a risk evaluation and mitigation approach aimed specifically at COTS-related issues in acquisition. CURE is most useful as a "front-end" analysis tool for predicting the areas where the effect of COTS components will be most prominent in a program. It can also be used as an overall improvement tool for organizations wishing to become more aware of COTS-related risk areas.

CURE consists of a detailed questionnaire completed by personnel in the program being evaluated and an evaluation report prepared by the SEI about the program. In addition, organizations can request optional follow-up visits to reassess high-priority risks, assess impact of program changes, identify new areas of risk.

"CURE came about, in part," says David Carney of the SEI, "because of our experience on 'red teams' for programs that were in trouble, and that were making heavy use of COTS components. We were going in and asking investigative questions to help find the source of the troubles in these programs. And we were asking the same questions over and over. So we distilled these questions into the questionnaire."

The questionnaire is used during on-site visits to an acquiring organization, a system developer/integrator, or both. One month before an on-site visit, the SEI sends the organization the 35-page questionnaire with questions targeted at decision makers—the program manager, lead architect, and system engineer—within a specific program or project. The questionnaire is intended to gather information about past programs as well as the current program.

Next, the evaluation team conducts on-site interviews with program personnel to analyze key topics in greater depth and discuss additional topics. "This is where we do the digging," says Carney. "The questionnaire includes a list of topics related to each question. This is to let the organization know the kinds of things we'll be looking for." See the sidebar accompanying this article for an example questionnaire item and the list of topics related to that item.

After the on site visit, the evaluation team produces an evaluation report for the organization. The report focuses on the COTS related risks in the program and suggests mitigations to those risks.

For each serious risk identified by the evaluation team, the report includes

  • a concise statement of the risk and a detailed review of the team's analysis
  • the sources of information that led to the analysis (for example, specific answers in the questionnaire or the portion of the interview)
  • the team's assessment of the criticality of the risk
  • some potential mitigations to the risk

What CURE Is Not

Other risk management tools, such as the SEI's SRE method, emphasize the need for risk management as a routine way of doing business; use of SRE can be an important catalyst for bringing about in an organization a philosophical change in attitude toward risks. CURE, by contrast, is diagnostic, and specific to a given program. It is not aimed at bringing a philosophical change, but at fixing a program before it gets out of control.

It is also important to note that the evaluation is not product oriented. "We don't come in and say, 'this is a good product' or, 'this is a good testing tool,' " says Carney. "We don't talk about specific products; we are process oriented." So rather than evaluating, for example, a specific testing approach and making a judgement about whether the organization should be using that approach, a CURE team would look for evidence that the approach has been used successfully in the past, asking questions such as, " What kind of program has this approach been used on? Was the program successful? Who else is using this approach?" Similarly, the questions that team might ask about specific products would be: "What is the evidence that this product behaves as advertised? What is the evidence that your contractor understands the product and has used it before?"

Results and Transition

CURE has been used successfully in several contracting and acquiring organizations, and in one case, both the contractor and acquirer on the same program. The SEI is currently working with one DOD agency on transition of CURE to the community and is seeking other organizations for a similar partnership.

CURE Questionnaire

Below is an excerpt from the CURE questionnaire. Note that in addition to seeking written responses, the questionnaire prompts interviewees to think about related topics that the SEI evaluation team will be probing in more detail during the site visit.

Describe the degree to which [the system] will require program-specific tailoring, extensions, or enhancements to COTS products.

Identify aspects of the system's design that are (or are expected to be) dependent on modifications to specific COTS products.

Potential discussion topics for on site interviews:

  • Strategy for product replacement when necessary
  • Decision factors indicating that modification is necessary
  • How the extent of modification was determined
  • Estimates for the cost and schedule of modifications

Find Us Here

Find us on Youtube  Find us on LinkedIn  Find us on twitter  Find us on Facebook

Share This Page

Share on Facebook  Send to your Twitter page  Save to del.ico.us  Save to LinkedIn  Digg this  Stumble this page.  Add to Technorati favorites  Save this page on your Google Home Page 

For more information

Contact Us

info@sei.cmu.edu

412-268-5800

Help us improve

Visitor feedback helps us continually improve our site.

Please tell us what you
think with this short
(< 5 minute) survey.