NEWS AT SEI

Software is everywhere, and developing it has become a major worldwide industry. We find software embedded, for example, in watches, coffee makers, cars, televisions, airplanes, telephones, reservation systems, and medical equipment. Software not only pervades a multitude of products, but also is an important corporate asset, and demand is increasing. Yet software projects are characterized by schedule and budget overruns and the delivery of unreliable and difficult-to-maintain software products.

Despite an exponential increase in the demand for and dependence on software, many software manufacturers exhibit unpredictable behavior. It is sometimes difficult to determine, for example, a software product’s release date, its features, the associated development costs, or the resulting product quality.

Uncertainty in the release date causes difficulties in planning product promotions, customer training, and maintenance support. Resource utilization across projects can become inefficient and difficult to manage when projects do not meet schedules. Customers have difficulties planning for the introduction of new software into their organizations when a scheduled release date is missed.

The exponential growth of the number of software products and releases suggests that end users will be exposed to more defects if the software industry is not able to reduce defect potentials and increase removal efficiencies at a similar rate. Competitive pressures related to feature content, time to market, and product quality will make product-release timing decisions both more important and more complex.

A Strategic Software-Release Methodology

Software-release decisions often have strategic implications because of the high costs of reversing the decision. Prospective losses also may arise long after the release decision has been made; for example, in cases where liability leads to lawsuits. Existing decision models for product release often do not account for market strategy. Ideally, release-decision criteria should align with corporate strategy.

A software-release decision can be seen from different perspectives:

• Maximizing behavior. A software-release decision is a tradeoff between early release to reap the benefits of an earlier market introduction and deferred release to enhance functionality or improve quality. If a software product is released too early, the software manufacturer incurs the post-release costs of later fixing failures. If a software product is released too late, the additional development cost and the opportunity cost of missing a market opportunity could be substantial. These two alternatives must be compared to determine which alternative maximizes economic value.
  
• Optimizing behavior. A release decision is affected by the difficulty of verifying the correct implementation of functional and non-functional requirements. How much testing is needed? Software manufacturers must find the optimal level of information because information has its price in cost and time. In practice, cost and time will constrain the ability to retrieve complete and reliable information, so this search for information should be considered as an economic activity. This leaves the software manufacturer with the problem of finding the optimal level of information where marginal value equals marginal costs and marginal yield is zero. This optimal level is difficult, if not impossible, to find.
• Satisficing behavior. Decision-making in the real world is often unstructured and normally involves various stakeholders who may have reasons to release a system or software product because of political or business pressures even though they know that it still contains defects. A study of spacecraft accidents, for example, reveals that, although system and software engineering were inadequate during development, management and organizational factors—including the diffusion of responsibility and authority, limited communication channels, and poor information flows—played a significant role [Leveson 04].
• Decision Implementation. A decision is considered successful if there is congruence between the expected outcome and the actual outcome. This definition sets requirements for decision implementation. In practice, there are many obstacles to the successful implementation of almost any decision, including
  
• the reduced importance of a decision once it is made and implemented
• the control of the outcome of a decision by stakeholders not involved in making it
• the development of new situations and problems that command the attention of the decision-makers once the choice has been implemented

Improving Strategic Software-Release Decisions

To investigate how to improve strategic software-release decision making, the SEI supported research into designing a release-decision methodology. This research reviewed the four perspectives detailed above from both a theoretical and an empirical point of view by studying practical examples. The results helped to frame a proposed release-decision methodology to address software-release decisions from different perspectives. The methodology, which consists of a defined set of practices, combines insights from economics, software management, and social psychology.

Studies in three different organizations validated the methodology. One participating organization is a leading global financial services company that provides financial services and products to retail and business markets. Services include insurance, pensions, occupational health and safety, asset management, investments, leasing, real estate, venture capital, and mortgage finance.

The research examined a project in this organization’s IT department, which develops custom systems for internal and external use. The initial estimate for the schedule was 10 months and for the pre-release cash outflows, €15M. Budgets were reserved for the technical infrastructure and post-release cash outflows for maintenance and exploitation. During the first months, the project encountered several setbacks: technical problems surfaced and the development budget turned out to be optimistic. Progress control was lacking, mainly through the absence of clearly defined milestones or quality gates. These problems increased, and in November 2001, the project was redefined. Both senior management and the marketing department exerted pressure on the product-development team to release the product as soon as possible. The team was, however, faced with an unstable product under test and had to use a veto several times to postpone a scheduled release date. When the product was released, uncertainty was high because many known problems were not resolved (although not considered critical), and the organization judged that continued testing would reveal more defects, including potentially critical ones, that could severely hamper the correct functioning and stability of the product.

After the product release, a special task force assumed responsibility for corrective-maintenance activities. This team needed more than a year to resolve the known and newly detected defects. Despite the original requirement to develop a maintainable product, the organization decided in 2004 to start a pre-study toward a new product to replace this product because corrective maintenance and functional enhancements proved difficult and costly. In other words, the early release of the product saved the organization additional testing costs, but the post-release maintenance cost turned out to be significantly higher than expected. A retrospective review of this project using the release-decision methodology enabled this organization to assess the project from a release decision point of view.

Figure 1 illustrates how the organization scored on the identified practices in the methodology. Lack of a product-development strategy (release definition) and lack of information as input to the decision-making process (release information) led to a poorly structured release-decision process without consensus among the stakeholders involved (release decision). Sufficient financial resources saved the organization in the short term by making it possible to patch the released software.

Figure 1: Radar Presentation of Case Study Results

These results from a practical setting indicate that this software-release decision methodology supports understanding, analysis, assessment, and improvement of the capability of software manufacturers in this problematic area. Research in software-manufacturing environments is under way to study the effects of applying the methodology at the start of projects to proactively aim for release-decision success.

Reference

[Leveson 04]
Leveson, N.G. “The Role of Software in Spacecraft Accidents.” AIAA Journal of Spacecraft and Rockets, 41, 4, 2004. Appendix: Release-Decision Methodology

Appendix: Release-Decision Methodology

In the release-decision methodology framework, four areas in the software-release decision-making process are distinguished, each addressing the process from a different perspective. A process area is defined as a cluster of related practices that, when performed collectively, achieve a set of goals considered important for establishing process capability in that area. Each process area consists of four relevant practices, describing what is to be accomplished but not how. Through this approach, the descriptions of practices provide the potential for interpretation and customization to the external market environment and to internal strategic and functional characteristics of a software-manufacturing organization.

Identified process areas are (see Figure 2):

1. Release Definition. Decision-making is mainly viewed from a quantitative perspective, assuming that information is nearly perfect—complete and reliable. It emphasizes the maximizing behavior approach with emphasis on mathematics, economics, and statistics. In software-release decisions, decision-making from a quantitative perspective is concerned with the definition and control of a product-development strategy—setting the managerial objectives with their priorities and ensuring that they are attainable. The availability of a product-development strategy enables the comparison and evaluation of different release alternatives, answering the question, “Which alternative maximizes economic value?”
2. Release Information. This process area is concerned with the search for alternatives during product development—for example, the identification and collection of information needed to compare and evaluate release alternatives. This search is derived from the formulated product-development strategy. Decision-making is also viewed from a quantitative perspective, but with the recognition that information is imperfect in the sense that not everything can be expressed in numbers and that information has its price in time and money. For this process, mathematics, economics, and statistics still play important roles, but the maximizing behavior approach is extended with an optimizing behavior approach: What is the optimal volume of information? Insufficient information increases uncertainty and hampers the decision-making process, whereas too much information is a waste of scarce resources. There is an optimum above which the cost for searching for more information exceeds the benefits.
3. Release Decision. Decision-making is viewed from a psychological, sociological, and socio-psychological perspective, addressing factors that influence individual and group behavior. It recognizes the imperfections of information and acknowledges that stakeholders involved in the choice may have different preferences with respect to the decision outcome. The challenge is to use a judgmental strategy to reach a decision that meets the formulated objectives and is agreeable to all stakeholders involved. The concept of optimizing behavior is extended with a satisficing behavior approach: Which outcome satisfies the needs of all stakeholders involved?
4. Release Implementation. Decision-making is viewed from an implementation perspective once a decision has been made and is implemented. This assumes that a successful decision requires follow-up and control. For software-release decisions, it is necessary to identify the factors that ensure congruence between the expected and the actual outcomes. The organization should evaluate the decision-making process and its outcome to promote organizational learning.

Figure 2: Release-Decision Methodology

About the Author

Hans Sassenburg works as a visiting scientist for the Software Engineering Institute in Europe. He is a part of the Software Engineering Process Management group. In 2002 he started doctoral-level research studying the area of software release decisions. This work was finished in January 2006.