A Framework for Software Product Line Practice, Version 5.0
Using Externally Available Software
The software for any system may be obtained in ways other than by developing it from scratch. Acquiring commercial, off-the-shelf (COTS) software, open source software, freeware (such as that from the Free Software Foundation), and Web-based services to populate service-oriented architectures are all promising options for organizations seeking to use already-existing software. Organizations do this in an effort to improve the cost-effectiveness and efficiency of their software development efforts [SEI 2007c, OSI 2007a, FSF 2007a].
Government software acquisitions also use the general terms commercial item (CI) and non-developmental item (NDI) to distinguish items, including software, that were developed by an organization other than the acquiring organization [FAR 2005a]. The term NDI is actually subsumed by the term CI.
The open source approach is the major alternative to acquiring proprietary software [OSI 2007a]. This approach is often more flexible than COTS but has distinct costs. Open source software is often accompanied by a license (discussed below) and sometimes comes with a formidable learning curve. Open source represents more than just a means of obtaining software: it is an important ingredient in the business model for a significant portion of the software market.
Many of the same opportunities and challenges apply to all the externally available software options (hereafter known as EAS). Other practice areas touch on EAS issues as well. Licensing issues are addressed in the "Developing an Acquisition Strategy" practice area. Integrating EAS with other software is a major challenge, as is testing it; see the "Software System Integration" and "Testing" practice areas, respectively. The "Mining Existing Assets" practice area covers the modification process of these items, once they're onboard.
The use of EAS components has mushroomed in large-scale system development. Communication infrastructures, network management software, databases, and a host of domain-specific utilities are available both as COTS and open source components. Many of these utilities are packaged as services and used to populate service-oriented architectures. For example, suppose that you're building an online auction Web site. You can find packages (or services) to manage the auctions, take credit cards for payment, and implement personalized search and notification functions for your customers. The list is endless. In many cases, entire architectures are now available in the marketplace. Using EAS is reported to cut costs, take advantage of common architectures, and enable large-scale reuse. However, when EAS is used in a system, you have far less control over how the components fit into the architecture and evolve. Using EAS introduces a new set of issues, concerns, and tradeoffs, many of which are listed in the "Practice Risks" section of this practice area.
Although specific practices vary, using EAS always necessitates the following steps:
Analyze for architectural compatibility: Determine the ways in which the software will fit into the architecture, paying particular attention to points of flexibility and variability, as well as areas that can't be compromised. Conversely, recognize that the availability of EAS may represent an advantage of such magnitude (over in-house development) that it might be worthwhile to modify the architecture to make a place for that software.
Understand the requirements of the organization: Some organizations have specific technical constraints or standards to which all software must conform. These constraints may rule out otherwise acceptable EAS. On the positive side, such constraints will quickly narrow the search for suitable candidates. In any case, the constraints need to be written down so they can be reviewed for completeness, be revisited occasionally for relevancy or revision, and be given to someone to guide the search for EAS.
Study the marketplace in detail: New products are being developed all the time; be alert to those that can be relevant to your product line. For large-scale efforts, it will pay to dedicate resources to keeping up-to-date on the new technologies, middleware products, and relevant EAS that are available in this rapidly changing field. Try following relevant standards groups and products conforming to standards. Determine subsets of the marketplace that are important, subscribe to newsgroups and lists that advertise new components, and maintain a list or database of potentially relevant components for when it's time to make a purchase decision. These activities are part of the "Technology Forecasting" practice area.
Develop requirements in a flexible manner: EAS is written to the vendor's expectations about the market, not to your organization's specifications. The vendors hope they meet your needs, but they probably won't do so exactly. Instead of traditional requirements that specify "must" and "should" needs, requirements for EAS articulate broad categories of needs and possible tradeoffs, in addition to the critical or nonnegotiable requirements. These flexible requirements can be used to narrow the field of candidates; if none are found and the fallback is in-house development, the requirements can be tightened as usual to drive that development. But the bottom line is that you need to keep an open mind about what you thought you needed, so you can take advantage of what's available.
Develop an approach for the evaluation of products and technologies: The approach should prescribe not only the evaluation steps but also the evaluation criteria. Include things such as price, vendor stability, the level of support required and provided, the ease of replacing the component with a competitor's product should the need arise, the simplicity of integration, and the ease of use. Be sure to include the important quality attributes that you expect the products to have: performance, security, reliability, and so forth.
Select viable products and technologies: Use the evaluation approach to qualify potential product candidates and make selections based on the evaluation criteria.
Buy the products: Buying software may not be simple. Organizational purchasing policies and guidelines must be followed. In some cases, you may want to test the products in prototypes before committing to a full purchase. In other cases, you can make a commitment immediately. Even when the EAS is free, following purchasing policies may prevent hasty decisions that are costly in terms of lost time and opportunities.
Integrate products into the architecture: Don't underestimate the task of integrating EAS with other EAS and with the rest of the system. Don't assume that EAS can simply be plugged into an existing architecture without modifications (of either the architecture or the product). Plan for adaptation with wrappers, middleware, or other software "glue." See the "Software System Integration" practice area for more advice on this topic.
Test the products and their configuration: The inclusion of EAS requires testing of the products' interfaces: remember that the components are essentially black boxes. Using EAS also requires testing any potential interactions with other components that may have an unpredictable impact. This is especially true for quality attribute requirements. Because EAS is developed by someone else according to a set of requirements unknown to you, quality attributes need to be considered specifically if factors such as security, performance, and availability are critical.
Manage the system on an ongoing basis: Monitor the current configuration of EAS, and scan the horizon for new products or potential replacements. Write guidelines for making decisions about upgrading and replacing components. Maintaining EAS means incorporating new releases into an existing set of core assets; therefore, maintenance requires attention to different upgrade cycles, potentially incompatible data sets, conflicting naming conventions, and new conflicts between different EAS. In addition, decide who will be the point of contact with the EAS vendors for issues, problems, and upgrades.
The discussion so far has related to using EAS in the form of components or services. It is also possible to adopt whole architectures and application frameworks off the shelf. A framework is a template for systems within a particular domain [Jacobson 1997a]. In the object-oriented world, a framework is often provided in the form of a set of class definitions. If a commercial or open source architecture is selected, it must be analyzed carefully to determine whether it is appropriate for the desired system. Perform an architecture evaluation just like you would for a homegrown architecture. The evaluation can likely be abbreviated, since many of the architecture's quality and behavioral attributes should already be known. An off-the-shelf architecture often admits the use of whole sets of available off-the-shelf components that are compatible with it.
Using an application-standard architecture brings about a subtle but significant change in focus. Instead of designing an architecture, specifying components, and then going to the marketplace to see what is available, evaluating the choices, and so on as described above, the marketplace is scoured for existing architectures and components and for systems built from both. The architecture is then selected or defined on the basis of the available EAS. From this perspective, EAS is used as product constraints that get factored into the requirements and early architectural activities and that greatly influence or even dictate the architecture [Albert 2002a].
Aspects Peculiar to Product Lines
EAS can populate the core asset base of a product line, or it can be relevant for the development of specific products in the product line. If they are core assets, EAS, like other product line assets, must be flexible enough to satisfy the variability needs inherent in a product line. Variability, then, must be added to the selection criteria. When selecting the appropriate EAS, you need to look at the variety of products it will be used in. Pay particular attention to their variation points and any requirements that can't be compromised.
Because of the central importance of the architecture for product lines and because of the need to fit a potentially large family of systems, a product line solution using EAS needs to be generalized, involving general-purpose integration mechanisms that span a number of potential products. As a result, the range of potentially qualifying products may be reduced, and the use of wrappers and middleware must focus on generalized solutions rather than on some of the more opportunistic solutions that may be appropriate for single-product systems. The evolution strategy for making updates is now more complex, because the range of affected systems is greater.
Application to Core Asset Development
EAS is certainly a viable choice to serve as core assets. Commercial components such as databases, graphical user interfaces (GUIs), graphics components, World Wide Web browsers, Java-based products, and other middleware products based on approaches such as .NET, remote method invocation (RMI), or their open source counterparts can represent an important part of the core asset infrastructure. Application- or domain-specific components or subsystems are also available. Because core assets need to be dependable over a long period of time, you'll need to pay attention to stability factors such as
- maturity and robustness of the products: Do they have a track record in the field, or will you be the first organization to use them?
- expected product update schedule: Will the vendor be issuing new releases every month? Every year? Every five years? Will the new versions be backward compatible, or will incorporating them require changes to other software in your system?
- interoperability with a wide variety of products: Do they work and play well with other assets, or do they make self-centered assumptions, such as the component that assumes it is the sole owner of the software's main timing loop?
- stability of the vendor: Is the vendor a trusted name in the field, or is it a new venture that is struggling to get off the ground and might be out of business in a year?
When evaluating EAS for adoption as core assets, consider the interfaces and the relevant protocols and standards to which the products conform. They will determine how well (or if) components will work within an infrastructure. Be aware that the mechanisms by which a product conforms to a standard may be idiosyncratic or inconsistent. In addition, since core assets can apply to a potentially large class of products, you should focus their testing on issues of generality and extensibility.
Licensing issues may be more complex when EAS is used as a core asset. Trivially, the license becomes a deployment item that may require user acceptance during the installation process. Ordinarily, a COTS license would be granted to use the product in multiple copies of a single system. But if the COTS product is a core asset, it will be used in multiple copies of many different systems, and the vendor is likely to want a different legal arrangement. There are many types of open source licenses. Even freeware may come with licensing restrictions. Services may require a fee-per-use that results in a prohibitively expensive proposition. The critical areas to consider are those where the free software interacts with proprietary software.
Application to Product Development
An individual product in a product line may contain product-specific code that applies only to that product. If that code corresponds to a full component, there is no reason why that component cannot be EAS. Of course, the economics of such a decision will have to be weighed; in that case, the decision will be the same as for EAS use in a single-system development. Beyond that, the technical factors for choosing EAS are the same as for a core asset.
COTS-based systems practices: The SEI developed a set of practices to help with COTS product and technology evaluation, COTS product acquisition and management, design and software engineering using COTS products, business case analysis, and COTS policy and planning [SEI 2007c].
Many of the guidelines apply not just to COTS but to other forms of EAS as well. For example, Carney suggests the following three-step approach for evaluating commercial products and technologies to see if they present a viable option for your development [Carney 1998b].
- Plan the evaluation.
- Define the problem, considering the entire context of the product line core assets (functional, technical, quality attributes, platform, resources, alternatives, and business issues).
- Define the outcomes of the evaluation.
- Assess the decision risk.
- Identify the decision maker.
- Identify the resources.
- Identify the stakeholders.
- Identify the alternatives.
- Assess the nature of the evaluation context.
- Design the evaluation instrument.
- Specify the evaluation criteria.
- Build a priority structure.
- Define the assessment approach.
- Select an aggregation technique.
- Select assessment techniques.
- Apply the evaluation instrument.
- Obtain products (for evaluation).
- Build a measurement infrastructure.
- Aggregate data.
- Form recommendations.
Open source software practices: The Open Source Initiative Web site [OSI 2007a] lists over 50 different licenses issued by a variety of organizations. The open source movement is still maturing with respect to specific practices.
Ineffective use of EAS can lead to schedule breakdowns during initial development as well as throughout the life cycle of a software product line. If a product is unsuitable to begin with or loses vendor support and becomes unsuitable, the result will be a sudden hole in the architecture, with a concomitant hole in the schedule while it is repaired. Major risks can be divided into issues relating to the EAS in question and issues relating to the vendor and vendor policies. Product-related risks include
unanticipated interactions: Unanticipated interactions may occur between the EAS and other components, resulting in a system that does not behave as intended. The mitigation to this risk is to study the EAS and its interactions thoroughly and make specific tests during integration testing. (See the "Testing" practice area.)
poor EAS quality: Similar to the previous risk, EAS without extensive track records pose the risk of failing to meet the reliability standards for the system. Failure to adequately qualify or test EAS can admit an unacceptable product into the system.
inappropriate product for the job: Failure to comprehensively qualify EAS for its intended use may result in the selection of EAS that fails to meet the behavioral requirements or (more likely) the quality requirements such as performance and security.
lack of adaptability: EAS often needs to be adapted to fit within an architecture and environment. Since EAS is usually packaged as a black box, this adaptation may be costly and result in unpredictable behavior, because the internal workings and assumptions of the EAS are unknown. Changing (or persuading the vendor to change) EAS is a risky move in any case, because the result will be an untested one-of-a-kind piece of software that won't track with future releases or other products in the vendor's inventory.
replacements for EAS: Replacements or substitutions for EAS will not represent one-for-one substitutions. Sometimes new EAS will be available that will perform most of the functions of an existing product. On the other hand, EAS may overlap partially with several components. Decisions about replacements need to be made within the context of the evaluation criteria for the selection of EAS.
Vendor-related risks include
inopportune updates to EAS: EAS updates will almost certainly not be synchronized with updates to the product line, and they may conflict with the technical direction of the product line. Make sure to develop an update strategy for making decisions about when to update EAS, as well as when to forego updates.
lack of support for EAS: The support for EAS will vary by product and vendor. Some products will be withdrawn by a vendor, and some vendors may go out of business. It is important to include specific evaluation criteria based on the vendor's stability, product strategy, support record, and update policy for specific EAS.
[Clements 2002c, p. 96]
Clements and Northrop provide an example of COTS product evaluation criteria.
[FSF 2007a] Likewise, the Free Software Foundation (www.fsf.org) discusses free software.
[Kenwood 2001a] Kenwood's paper provides a business case study of open source software conducted by the Mitre corporation for the U.S. Army.
[OSI 2007a] The Open Source Initiative (www.opensource.org) is a good resource for learning more about using open source software.
[SEI 2007c] The SEI's CBS Web site provides a host of papers and monographs on COTS product usage and pointers to further resources, including courseware. The "Gotchas of COTS for Acquisition Managers: What You Don't Know Can Hurt You" presentation (available at http://www.sei.cmu.edu/cbs/cbs_slides/stc98/Gotchas/index.htm) is a good myth-shattering introduction to the issues.
[Wallnau 2002a] The book by Wallnau, Hissam, and Seacord is a comprehensive treatment of the integration of commercial components in a system.
[Wheeler 2005a] Wheeler's paper covers a range of issues associated with using EAS.