NEWS AT SEI
This article was originally published in News at SEI on: March 1, 2001
In this and the next few columns, I discuss the future of software engineering. This column focuses on trends in application programming, particularly as they concern quality. In subsequent columns, I address programming skills, trends in systems programming, and the implications of these trends for software engineering in general. While the positions I take and the opinions I express are likely to be controversial, my intent is to stir up some debate and hopefully to shed light on what I believe are important issues. Also, as is true in all of these columns, the opinions I express are entirely my own.
Some 50 years ago when I finished graduate school, I started to work with computers. For almost all of the intervening time, people have been dreaming up new and better ways to use computing devices and systems. While the volume of application development has grown steadily, so has the list of applications that people want to develop. It seems that the more programs we write, the more we understand what we need, and the more programs we add to the application-development backlog. So far, the rate of growth in application demand has continued to accelerate.
As economists say, unsustainable trends are unsustainable. However, to date, there is no sign that this growth in demand is slowing. I suspect that it will continue for the foreseeable future, though we will certainly see changes in the types of application programs. The reason for this growth is that application programming is a very effective way to meet many kinds of human needs. As long as people continue devising new and cleverer ways to work and to play, we can expect the growth in computer applications to continue. The more we learn about computing systems and the more problems we solve, the better we understand how to address more complex and interesting problems. Therefore, because human ingenuity appears to be unlimited, the number of programs needed in the world is also essentially unlimited. This means that the principal limitation on the expanding use of computing systems is our ability to find enough skilled people to meet the demands.
In discussing these topics, I break the software world into two broad categories: applications and systems. I won’t try to clearly define the dividing line between these categories because that line is both indistinct and changing. By "application programming," I refer to solving human problems with computing systems. Here the focus is on defining the problem and figuring out how to solve it with an automated system. Systems programming focuses on how to provide automated systems, aids, tools, and facilities to help people produce and run application programs. In looking toward the future in these areas, I believe that the two most significant issues concern software quality and the demand for skilled people. I address application software quality first.
Application Program Quality
The quality of application software today is spotty at best. A few programs are of high quality and many are downright bad. The discussion of application quality has two principal parts. First, computers are being used for progressively more critical business applications. This means that, even at current quality levels, the impact of software defects will grow. This implies that the demand for high-quality application software will also grow.
To date, people buy and use software without apparent concern for their quality. While they complain about quality problems, software quality has not yet become a significant acquisition consideration. Until it is, we cannot expect suppliers to measure and manage the quality of their products. However, as has been amply demonstrated in other fields, when quality is not measured and managed, it is generally poor.
When the public gets concerned about quality, these attitudes will quickly change. It will not take many high-profile disasters to cause executives to worry. Then, they are likely to demand quality guarantees and warranties from their suppliers before they entrust their businesses to new computing systems. When customers cannot be assured of the quality of a software package, they will either go to a competitor or forgo the application entirely and continue using prior methods. This would not be because automated methods would not have been helpful, but simply because the risk of failure would be too high.
Growing Program Complexity
The second part of the program quality discussion concerns size and complexity. The size and complexity of application programs is increasing. The size data in Figure 1 show just how fast this growth has been. The IBM size measures are from my personal recollection, the Microsoft NT size data are from published reports, and the spacecraft size data are from Barry Boehm [Boehm 81, Zachary 94]. The TV data are for the embedded code in television sets and are from a talk by Hans Aerts and others at the European SEPG conference in June 2000. According to my prior definitions, the IBM and Microsoft products are system software while the spacecraft and TV programs are application software.
These data show that the size of the software required for various kinds of systems and applications has been growing exponentially for the past 40 years. The trend line in the center of the chart shows a compound growth rate of ten times every five years. This is the same as Moore’s law for the growth in the number of semiconductors on a chip, or a doubling every 18 months.
While the size growth of system software has been phenomenal, it appears to be slowing, at least from the appearance of the IBM and NT data in Figure 1. I discuss these systems software trends in a later column. The critical point from an application quality point of view is that the growth trend for applications software appears to be continuing.
The Defect Content of Programs
Assuming that the same methods are used, the number of defects in a program increases linearly with its size. However, the rate at which application program users experience problems is largely determined by the number of defects in a program rather than their density. Therefore, even though the defect density may stay about the same or even improve, merely producing larger programs with the same methods will produce progressively less reliable systems. So, either the quality of future application software must improve—at least in step with the increasing sensitivity of the applications—or businesses must limit their use of computing systems to less critical applications.
Figure 1: Program Size Growth
To see why program reliability depends on defect numbers instead of defect density, consider an example. A 200 KLOC (thousand lines of code) program with 5 undetected defects per KLOC would have 1,000 defects. If you replaced this program with an enhanced program with 2,000 KLOC and the same defect density, it would have 10,000 defects. Assuming that the users followed a similar usage cycle with the new application, they would exercise the larger program at about the same rate as the previous one. While this would presumably require a faster computer, the users would be exposed to ten times as many defects in the same amount of time. This, of course, assumes that the users actually used many of the new program’s enhanced functions. If they did not, they presumably would not have needed the new program.
An Application Quality Example
Oil exploration companies use highly sophisticated programs to analyze seismic data. These programs all use the same mathematical methods and should give identical results when run with identical data. While the programs are proprietary to each exploration company, the proprietary parts of these programs concern how they process enormous volumes of data. This is important because the volume of seismic data to be analyzed is often in the terabyte range.
A few years ago, Les Hatton persuaded several oil-exploration companies to give him copies of nine such programs [Hatton 94]. He also obtained a seismic exploration dataset and ran each of these nine programs with the identical data. The results are shown in Figure 2. Here, the range of calculated values is shown for several consecutive program iterations. As you can see, this range generally increased with the number of cycles, and after a few runs, it reached 100%. When one of these companies was told about some of the conditions under which its program gave unusual results, the programmers found and corrected the mistakes, and the program’s next results agreed with the other programs.
Figure 2: Growth in Seismic Program Uncertainty
The results produced by these oil-exploration programs contained errors of up to 100%, and these results were used to make multi-million-dollar decisions on where to drill oil wells. Based on this study, it appears that these programs provided little better guidance than throwing dice. I am not picking on these programs as particularly poor examples. Their quality appears to be typical of many application programs.
The Quality Problem
In discussing the quality of application programs, we need to consider the fact that defective programs run. That is, when engineers produce a program and then run extensive tests, they generally can get it to work. Unfortunately, unless the tests were comprehensive, the tested program will likely contain a great many defects.
Any testing process can only identify and fix the defects encountered in running those specific tests. This is because many program defects are sensitive to the program’s state, the data values used, the system configuration, and the operating conditions. Because the number of possible combinations of these conditions is very large, even for relatively simple programs, extensive testing cannot find all of the defects.
Since the size of application programs will continue to increase, we need to consider another question: will we be able to test these programs? That is, how well does the testing process scale up with program size? Unfortunately, the answer is not encouraging. As programs get larger, the number of possible program conditions increases exponentially. This has two related consequences.
- The number of tests required to achieve any given level of test coverage increases exponentially with program size.
- The time it takes to find and fix each program defect increases somewhere between linearly and exponentially with program size.
The inescapable conclusion is that the testing process will not scale up with program size. Since the quality of today’s programs is marginal and the demand for quality is increasing, current software quality practices will not be adequate in the future.
The Impact of Poor Quality
As the cost of application mistakes grows and as these mistakes increasingly impact business performance, application program quality will become progressively more important. While this will come as a shock to many in the software community, it will be a positive development. The reason is that suppliers will not generally pay attention to quality until their customers start to demand it. When software quality becomes an important economic consideration for businesses, we can expect software organizations to give it much higher priority.
While one could hope that the software industry would recognize the benefits of quality work before they are forced to, the signs are not encouraging. However, improved product quality would be in the industry’s best interests. It would mean increased opportunities for computing systems and increased demand for the suppliers’ products. This would also mean continued growth in the demand for skilled software professionals.
In the next column, I discuss the need for application programming skills and how this need is directly related to the quality problem. Following that, I discuss related trends in systems programming and the implications of these trends for software engineering.
In writing papers and columns, I make a practice of asking associates to review early drafts. For this column, I particularly appreciate the helpful comments and suggestions of Sholom Cohen, Noopur Davis, Alan Koch, Don McAndrews, Julia Mullaney, Bill Peterson, and Marsha Pomeroy-Huff.
In closing, an invitation to readers
In these columns, I discuss software issues and the impact of quality and process on engineers and their organizations. However, I am most interested in addressing issues that you feel are important. So, please drop me a note at email@example.com with your comments, questions, or suggestions. I will read your notes and consider them in planning future columns.
Thanks for your attention and please stay tuned in.
[Boehm 81] Boehm, Barry. Software Engineering Economics. Englewood Cliffs, NJ: Prentice-Hall, 1981.
[Hatton 94] Hatton, Les. "How Accurate is Scientific Software?" IEEE Transactions on Software Engineering, Vol. 20, No. 10 (October 1994): 785-797.
[Zachary 94] Zachary, G. Pascal. Showstopper! New York: The Free Press, 1994.
About the Author
Watts S. Humphrey founded the Software Process Program at the SEI. He is a fellow of the institute and is a research scientist on its staff. From 1959 to 1986, he was associated with IBM Corporation, where he was director of programming quality and process. His publications include many technical papers and six books. His most recent books are Managing the Software Process (1989), A Discipline for Software Engineering (1995), Managing Technical People (1996), and Introduction to the Personal Software Process (1997). He holds five U.S. patents. He is a member of the Association for Computing Machinery, a fellow of the Institute for Electrical and Electronics Engineers, and a past member of the Malcolm Baldrige National Quality Award Board of Examiners. He holds a BS in physics from the University of Chicago, an MS in physics from the Illinois Institute of Technology, and an MBA from the University of Chicago.
The views expressed in this article are the author's only and do not represent directly or imply any official position or view of the Software Engineering Institute or Carnegie Mellon University. This article is intended to stimulate further discussion about this topic.