A Framework for Software Product Line Practice, Version 5.0

Make/Buy/Mine/Commission Analysis

In "A Note on Terminology", we pointed out that software enters an organization in one of three ways: it can be

built in-house
purchased from a commercial vendor–either whole (as in a commercial off-the-shelf [COTS] component) or as licensed rights to use the software (as in open source software or a Web-based service)
commissioned through a third party to be built especially for the organization

Software that is built in-house can actually be constructed anew or mined from software already in the organization for use in a new effort. Every piece of software that is part of a development effort arrived as the result of an all-encompassing four-way choice that we call "make/buy/mine/commission." Organizations that build software systems must all make this choice, but they usually lack a conscious rationale for their selection. The purpose of this practice area is to both underscore the necessity of making a conscious and reasoned choice and describe some of the analyses that can help an organization make the right choice.

Techniques from the discipline of decision analysis apply well here. Decision analysis is the process of applying analytical methods to decision making in situations where there is uncertainty, multiple conflicting objectives, or dynamic change [Clemen 1991a, Hammond 1999a]. Such a situation exists when an organization is trying to choose the best way to obtain software and must, as a result, weigh several business, technical, and political factors. In such a case, making the decision requires both qualitative and quantitative analysis.

Sometimes the decision is straightforward. For some organizations, all the software assets are developed in-house for proprietary (or political) reasons. For other organizations, all the assets are commissioned because of organizational policy or because of unique requirements and a lack of in-house development resources. (In the U.S., most government organizations, such as the DoD, fit into this category.) If no other organization but yours has the necessary domain expertise in a component's realm, "buying" and "commissioning" are not viable alternatives. Conversely, if you have neither the skill nor the history to build a component, "making" and "mining" are not going to work. More commonly, however, some of the software will be built from scratch, some will be mined, some will be purchased on the open market, and some will be commissioned.

The make/buy/mine/commission decision for software is based on strategic factors such as the cost, schedule, staff availability, and expected quality and fitness of purpose that each alternative offers. For example, mining is valuable only when a project that uses the mined assets can be completed on time at a lower cost and produce a capability equal to or greater than that of comparable assets obtained through other means. Any calculation of reuse cost should include the total cost of use over the lifetime of the new product or products, not just the cost of mining/restoring a particular set of assets. In practice, improvements on one of the scales (at a cost to the others) may produce a significant tactical advantage. For example, if mining and restoration gain time but lose cost and functionality (relative to building from scratch), they could still provide a significant advantage if time to market is a primary driver. The direct cost of each alternative is a factor, but so is the opportunity cost: what could your staff be doing instead of building a component if you buy or commission it? If you developed it in-house, what could you do with the funds you would save by not employing an expensive contractor? If the staff is underused, the "make" and "mine" options get more weight. If the staff is overused and the schedule is tight, the "buy" and "commission" options get more weight. Other factors influence the decision when a service-oriented approach is used. For example, are suitable service providers available to offer the necessary types of services in a form that is compatible with the product line architecture? Will it be possible to sign a service level agreement (SLA) with particular vendors to guarantee the quality of service (QoS) needed for the product line?

When analyzing the

"buy" option, refer to the "Using Externally Available Software" practice area
"make" and "mine" options, practices in the "Architecture Evaluation, "Mining Existing Assets," and "Component Development" practice areas will help
"commission" option, you must rely on the contractor's past performance for insight into whether it will be able to deliver as promised. Here, vendor reliability and stability are key, as for a COTS component; see the "Developing an Acquisition Strategy" practice area.

The make/buy/mine/commission decision for software should also be based on a frank assessment of an organization's own capabilities. Establishing working relationships with contractors can be tricky, and the legalities of drawing up an ironclad contract can be formidable for those without experience. If your organization does not have a standing legal department for which such relationships are pro forma, you might be better off on the "make/mine" side of the equation–that is, unless building such a capability for the future makes strategic sense for your organization. Conversely, if your development capabilities are weak, letting others tackle the technical complexities may be the best approach–again, unless you are trying to improve your organization's development skills or domain expertise. In either case, start your venture into the unknown with small steps, using the newest approaches for only the least critical components. In addition, have contingency plans to handle major missteps along the way.

Also keep in mind that the four options are not always mutually exclusive-for a given asset, you might do a little of each; for example

A component can be built largely from scratch but with some percentage derived from a legacy system.
Commissioned software is sometimes based at least partially on legacy assets.
How much customization or alteration of a COTS system is allowed before that system falls into one of the other three categories?
A software component could be purchased as a service on an ongoing basis from a service provider.

Since the make/buy/mine/commission decision analysis can be complex, making the decision consciously and documenting the rationale is a good idea.

Aspects Peculiar to Product Lines

The analysis approach for a product line is similar to that for single systems but with two main differences. When a product line is involved, you must perform the analysis (1) while constantly considering the satisfaction of the product line's production constraints and strategy and (2) using different weighting factors because of the reasons described below:

Costlier options that would be ruled out for single systems may be acceptable for product lines because the cost can be amortized over a number of products. For example, in a single system, you might be willing to use a COTS component because it would be cheaper than building your own. However, in a product line, you may be willing to pay the higher cost of in-house development, so your entire group of products is not held hostage by a vendor's version release and upgrade schedule. Similarly, certain licensing arrangements might make the use of externally available software reasonable for a single system but prohibitively expensive for a product line.
The "make," "mine," and "commission" options are usually more expensive because if the asset is destined to join the core asset base, it will have to be robust enough to be reusable across the entire family of products.
The "buy" option, on the other hand, may not be any more expensive than for a single-system case (licensing agreements notwithstanding) because COTS components tend to be built for generic usage. Of course, you still have to find an off-the-shelf component in the commercial marketplace that has the required variability and quality. Searching for and qualifying a component that's suitable for your product line may be more expensive than if it were going to be used in only one system.
Considering external services (provided by outside organizations) as product line assets impacts the "buy" option because what you're buying is the right for your system to use those services. You will have to determine how many external services are to be used, the ongoing costs involved, and whether some of those costs will be passed on to the end users.
Product lines tend to be built on a legacy foundation; the realization that a company is building many similar products is often the impetus for the product line approach. Hence, "mining" is a more viable alternative in a product line because of the likely existence of a rich legacy base.

The decision analysis for a product line must also look further into the future for multiple products that will be spawned, each having its own lifetime. In a product line, much more is riding on the decisions about how and where to obtain software. Analysis is essential, and rigorous analysis is warranted.

Application to Core Asset Development

Because all core assets have to come from somewhere, make/buy/mine/commission analysis is at the root of core asset development, and the analysis applies to non-software assets as well as software assets. The possibility of commissioning entire swaths of product line development (such as domain analysis, scoping analysis, market analysis, requirements engineering, and testing) should be considered. Requirements, architectures, and designs–as we describe in the "Mining Existing Assets" practice area–are superb candidates for recovery and rehabilitation from previous systems. The criteria to consider during the analysis include

quality
the cost (including the opportunity cost, that is, the loss in terms of the productive work the staff could have been doing instead)
alignment with the product line requirements
alignment with the product line architecture
sufficient flexibility for supporting requisite variation among the products in the product line
maintainability
the schedule
the ability of your organization to prosecute each of the four options successfully
the availability of core assets as services

The source of the product line architecture must be chosen prior to, or at least concurrent with, deciding how to obtain individual component or service assets. The architecture, by definition, specifies constraints for component and service assets-constraints that must be factored into any analysis about how those assets will be obtained.

Application to Product Development

Individual products in the product line may need additional software not contained in the core asset base. The same decision analysis process should be used to determine how to obtain this software. The criteria may be less demanding if the component or service in question will not be a candidate for inclusion in the core asset base. Although the component or service would still need to align with the product line architecture, it wouldn't have to offer the variability needed to support other products. On the other hand, the cost of obtaining the component or service may be more of a consideration because it has to be absorbed by a single product rather than being amortized over a family of products.

For example, if there isn't a human-computer interface (HCI) component or subsystem in the core asset base and a specific product needs one, the product builder can select the HCI component that is optimal for the needs of that particular product and the allowable budget. On the other hand, if an HCI component would be a viable candidate for addition to the core asset base, the decision analysis would rely on the same criteria used for core asset development, as described above. Furthermore, if external services are to be used within an individual product, the cost of using these services must be considered.

Example Practices

Breadth-first analysis of options: One approach is to adopt a breadth-first strategy in which each of the four options for obtaining software is considered and explored before any is eliminated. For some options, the consideration may be trivial. For example, if organizational policy dictates the commissioning of all software systems, the "make" choice requires no consideration: it's simply not an option.

Example questions for analyzing the make/buy/mine/commission decision are given below. First, there is an umbrella set of questions for the breadth-first strategy, followed by four sets of specific questions–one for each of the four options. Since each organization is unique, these sample questions should be viewed as a starter set that you can customize and augment as necessary.

Umbrella questions that follow the breadth-first strategy:

What are the time constraints for obtaining the asset? What external factors drive them? How reliable are the external predictions?
How well-defined is the asset that must be obtained? Have the product line requirements been defined? Have the required software commonalities and variabilities of all the products in the product line been defined? Has the system architecture into which the software will fit been defined completely? (If any of these items are not well-defined, it will be hard to hand off the responsibility for developing the asset, either software or non-software, to an outside party.)
Is there a market for the asset that is separate from the market for the product line? (If not, you probably won't find the asset on the open market as a COTS product.)
How flexible are the product line requirements and to what extent might the availability of a specific asset impact the requirements?

After this initial analysis, the choices may be narrowed to two, three, or even one of the four major options. A more detailed analysis for each option still in the running could include questions such as the following:

Sample questions for the "make" option:

Are developers with the necessary expertise available?
How would those developers be used if they were not on this project?
If additional personnel need to be hired, will they be available within the needed time frame?
How successful has the organization been in developing similar assets?
What specific flexibilities are gained by developing products in-house as opposed to purchasing them?
What development tools and environments are available? Are they suitable? How skilled is the targeted workforce in their use?
What are the costs of development tools and training, if needed?
What are the other specific costs of developing the asset in-house?
What are the other specific benefits of developing the asset in-house?

Sample questions for the "buy" option:

What assets are externally available either as COTS software, open source software, freeware, or services?
How well does the externally available software conform to the product line architecture?
How closely does the externally available software satisfy the product line requirements?
Is it possible to establish an SLA with the vendors of the software services that will meet the needs of the product line requirements?
Are small changes in the externally available software a viable option?
Is source code available with the software? What documentation comes with the software?
What are the integration challenges?
What, if any, redistribution rights are purchased with the externally available software?
What other specific costs are associated with purchasing the software? Is there an ongoing cost for the use of the software services?
What other specific benefits are associated with purchasing the software or the service?
How stable are the vendors?
How often are upgrades produced? How relevant are the upgrades to the product line?
How responsive is the vendor to user requests for improvements?
How strong is the vendor's technical support for the software or service?

Sample questions for the "mine" option:

What legacy systems are available to mine assets from?
How close is the functionality of the legacy software to the required functionality?
What is the defect track record for the software and non-software assets?
How well are the legacy systems documented?
What mining strategies are appropriate? How expensive are they?
What experience does the organization have in mining assets?
What mining tools are available? Are they appropriate? How skilled is the workforce in their use?
What are the costs of mining tools and training, if needed?
What other specific costs are associated with mining the asset?
What other specific benefits are associated with mining the asset?
What changes must be made to the legacy asset during the mining? What are their costs and risks?
What types of non-code assets are available to be mined? What, if any, modifications or additions will they need?

Sample questions for the "commission" option:

What contractors are available to develop the asset?
What is the track record of the contractor in terms of schedule and budget?
Is the acquiring organization skilled in supervising contracted work?
Are the requirements defined well enough to subcontract the asset's development?
Are interface specifications well-defined and stable?
What experience does the contractor have with the principles of product line development?
Who needs to own the asset? Who maintains it?
What other specific costs are associated with commissioning the asset?
What other specific benefits are associated with commissioning the asset?
What are the costs of maintaining the commissioned asset?
Does commissioning the asset involve divulging to the contractor any technology or information that, in the acquiring organization's best interest, should not be revealed?

These question sets are meant to be starter sets. Weigh these factors according to your organization's needs and experience, and augment them with factors of your own.

OAR/SMART: The SEI Options Analysis for Reengineering (OAR) approach is used to make decisions on mining assets for product lines and to address the specific mining questions outlined above [Smith 2002a]. OAR is explained in more detail in the "Mining Existing Assets" practice area. Similarly, the SEI Service-Oriented Migration and Reuse Technique (SMART) helps organizations analyze a legacy system to determine whether its functionality, or subsets of it, can be reasonably exposed as services in a service-oriented architecture (SOA) [Lewis 2005a]. This technique may be useful if a service-oriented approach is used within a product line. Economic models such as SEI SIMPLE (discussed in the example practices for the "Building a Business Case" practice area) can help answer questions regarding the relative costs or cost savings of each option.

In some cases, the "commission" option may involve commissioning a supplier to mine assets from an existing system. Bergey and colleagues suggest that under these circumstances a specific set of practices should be included to

determine that potential suppliers have the needed skills, mining experience, and tool support
ensure that mining recommendations are based on solid technical analysis
conduct a mining exercise to validate some aspect of the supplier's mining process and ability to produce product line assets
secure visibility into the planning, mining analysis, and decision-making process of a potential supplier [Bergey 2004a]

Tool support: Although spreadsheets are the most common tool used in decision analysis, tools that are more sophisticated are also available on personal computers. For example, TreeAge Software offers tools called DATA and DATA Interactive [TreeAge 1999a]. With DATA, an analyst can build sophisticated decision models as decision trees, influence diagrams, or Markov processes and then analyze them using analysis tools such as sensitivity analysis, cohort simulation, and Monte Carlo simulation. A companion product, DATA Interactive, provides a model customization and delivery system for accessing and analyzing the decision models created in DATA. These tools require knowledge about decision analysis principles, the problem domain, and the tools' functionality for building and analyzing models.

Practice Risks

Do not confuse the risks of this practice area with those of the practice areas associated with each of the four options (i.e., "Component Development," "Using Externally Available Software," "Mining Existing Assets," and "Developing an Acquisition Strategy"). The risks of this practice area are related to choosing among those options.

The major risk specific to this practice area is, of course, choosing the wrong option, which can result in inappropriate assets and undue direct (or opportunity) costs. This risk could occur for the following reasons:

There are not enough data on which to base decisions.
The data are inaccurate.
Political pressures force the decision.
There is no process for the decision analysis.
Schedule pressure precludes conducting a thorough analysis.
Not all the options are considered. Normally, all four options should be kept on the table for the initial analysis steps.
There is no established or documented product line architecture to factor into the decision.
There are poor estimates of the probabilities of projected outcomes.
The product line requirements have not been defined adequately.
Hidden interactions between alternative choices exist and may produce unpredictable consequences.
There is a lack of documentation to support decision analysis.
The differences in business interests between the acquiring organization and the supplier are not adequately factored into the decision-making process when choosing the commission option.