NEWS AT SEI
This article was originally published in News at SEI on: March 1, 2000
In this Roundtable interview, we present a wide-ranging discussion of open source software—its current popularity, its advantages and disadvantages for software developers and users, the implications for computer security, and whether it is appropriate for Department of Defense (DoD) systems.
The participants are Chuck Weinstock, Pat Place, Scott Hissam, and Dan Plakosh, all members of the SEI’s COTS-Based Systems Initiative, and Jed Pickel of the CERT Coordination Center at the SEI. The moderator is Bill Thomas, editor of news@sei.
Why Do Research on Open Source Software?
Bill Thomas: Why do you think that open source software warrants research at this time?
Chuck Weinstock: One reason is that it appears that the community is treating open source software as the next silver bullet. We all know that silver bullets very seldom find their target, and the community moves on to the next silver bullet.
Pat Place: I would add that there is a substantial amount of software available these days that is open source. If you are interested in building systems out of existing components—be they open source or any other form of source—you need to understand at least the risks as well as the benefits of doing so. If we can say anything that helps, then I think that's a good thing.
Scott Hissam: The phenomenon that is happening with the Linux environment is getting everybody's attention to open source software—more attention than has ever been paid to it before, at least in the media. People are enamored and believe that Linux is a successful, stable development environment, and that somehow every piece of open source software that they get is going to be just as stable and just as reliable as the Linux platform—if you believe that it is as stable and reliable as it is touted to be in the press.
Jed Pickel: Open source has been around for a long time—probably more than 20 years. I think one of the reasons why it's getting so much attention now is because commercial interests are developing it. That's why we’re seeing the media interest.
Is There a Community of Developers?
Dan Plakosh: You can release your source code, but I’m not sure people really know what to do with it. I released open source software two years ago and I’ve had very few people dive into developing it.
Place: It gets even worse when you get something that is hundreds of thousands of lines or a million lines. The historical aspect is interesting because you can look back at the history of programs that were open source, or close to open source, with lots of people helping and providing fixes. You can start to see what was successful about them and what is different about what is becoming the current open source movement, which I honestly believe is going to lead to disaster. I can provide anecdotal evidence of examples where you've got something like a tcsh [an expanded version of the original C shell for the UNIX operating system] which is not that complicated a program, but has features and peculiarities that are so weird that you'd never even want them. And yet somebody has said, "Oh, I'll just go and stick this thing into the system." For example, in tcsh, if you have time displayed as part of your prompt and it happens to hit the hour, it'll go "ding" instead of printing the time. I mean, this is insanity: feature upon feature upon feature that leads to code that's got more junk in it than you can possibly be interested in. It ends up becoming ultimately unmaintainable code.
Hissam: But I would say that the tcsh example that you gave is an unbounded development activity that nobody really paid attention to. Nobody really cared about it and that's why it got unwieldy. Not every piece of open source software is developed in that way.
Place: That's exactly true. I think that's the key to the difference between those things that have been and will be successful and those things that will not be successful. Somebody or some very small group of people have a very clear idea as to what that system is going to be, what it's going to do, and how it's going to be architected. And they keep it that way.
Weinstock: Some people refer to those people as the "arbiters of good taste."
Place: That's the phrase that was used primarily about the original UNIX developers. They were arbiters of good taste. With all of the stuff that people from all the universities shipped to them back in the mid-1970s and early ’80s, they decided what went into the source and what was not in the source. For the longest time with Linux, Linus Torvalds [the Finnish graduate student who created the original Linux operating system] was the person who did that. He had a vision as to what it was going to be. That seems to be drifting out. Linux is perhaps losing some of that arbiter-of-good-taste quality.
Pickel: On your tcsh example: there are plenty of examples of closed source software having very similar things. For example, one commercial product is such that if you hit certain key sequences, you can end up with a flight simulator—which is a little bit different from tcsh beeping at the end of a line. The difference, though, is that should the community come across that item in tcsh, and feel like it needs to be removed, and there are enough people who agree with that, then it would be removed from tcsh. You can easily go and change that if it bothered you enough.
Place: Absolutely. I've done that because I wanted tcsh to be as small as possible and I’ve used it as small as possible.
Hissam: So you removed a whole bunch of things out of tcsh that you didn't like. Right? Now let’s say the next version of tcsh comes out and you want to adopt it.
Place: I have developed the version of the shell that has the capabilities I need. If anything does come out in tcsh that I'm interested in, I might take that as a patch file and patch my source with those changes. But I'm not taking all their stuff again.
Weinstock: You now own the problem.
Place: Yes, absolutely. I willingly accept that. Of course, the advantage is that it was open source. I could choose to take on the risk and build something that was what I wanted.
Who Has Accountability for Open Source Software?
Thomas: It seems that no one has any accountability with open source software. It’s strictly "buyer beware."
Pickel: I would disagree with that. Let's go back to the tcsh example again, because I think that's a good one in that a person maintains it and is accountable for listening to the users. Pat didn't speak up in this case. He decided to split off his own version and now he's accountable for that.
Someone wants the X widget or the Y widget, so they just go and put that fix in, and you get this loss of sanity.
Place: If you look at the actual source code, you'll see all of these different names of people who've added this and added that. The risk I see with open source is that all of these features are getting thrown into a basically good system. Someone wants the X widget or the Y widget, so they just go and put that fix in and you get this loss of a sense of stability of the project—this loss of sanity. tcsh going "ding" is kind of stupid. It goes to my notion of good taste; it's below the cut line.
Pickel: What you're describing is an example of open source working in the most optimal sense in that there are people who have different goals from a project than you do. You decide to split off your own version; that's open source working right there.
Hissam: It depends on what your own goals are. If your own goals are to keep up with the latest and greatest, then him vectoring off his own version—he's stuck.
Place: That's disaster if you want to keep up with the latest.
Plakosh: I don't think people develop open source software with the intent that people will take it and go off and build their own products from it. I think it's more so that people will contribute to mature whatever piece of software that they're doing.
Place: The freedom for anyone to make a change leads to the fact that the product will never mature because it will always be in a state of flux.
Plakosh: There is not the freedom for anybody to make a change. I really think you're looking at isolated cases. For example, take Linux. The majority of people who use Linux never look at the source. In the majority of most open source products, I would bet that people do not look at the source. They don't care. They don't recompile it. They don't want to have anything to do with it. It's only the people who are working toward the development of Linux who are looking at the source, or occasionally someone who has the in-depth knowledge finds a bug and looks at the source. They may fix it but they may also submit it to one of the Linux working groups to have it corrected.
What does it mean to be accountable?
Hissam: Let’s go back to the earlier question about accountability. We disagreed about whether anyone is accountable. What does it mean to be accountable? It means that there's liability on the part of somebody.
Place: I wouldn't even go as far as that. Dan has released source. He's put his name up saying, "I think this is a good piece of source." In an open source project, other than a couple of special cases, there's a substantial body of existing code that gets released and then people can work on it, rather than working on stuff from scratch. I see a potential split there. Take Linux. Who is accountable for Linux these days? Does Linus put his name on it saying, "I think this is all good source code"? I don't think so anymore.
Hissam: No. The worst thing that can happen to the people who are "accountable" for Linux is that the world turns their back on it. But concerning accountability: I think the answer is that no one is accountable outright and I think it is buyer beware.
Weinstock: So that's why people go to places like [Linux vendor] Red Hat instead of just downloading it off the Web.
Hissam: Because they want to hand money to somebody. They want to be able to say, "Give me this and give me that."
Weinstock: Red Hat also sells support. You can go just buy a Red Hat CD and you get nothing with it other than the CD. But you can also go to them and get support for Linux.
Pickel: That's how Linux has made it into the corporate world: doing support.
Plakosh: I've dealt with support before and support is not typically all that it’s cracked up to be. Support is usually geared toward people who have problems in reading the documentation or who don't understand things. Linux got into the commercial domain mainly due to the attractiveness of it being free and being somewhat reliable.
What Are the Benefits and Drawbacks of Developing Open Source?
Thomas: Let's backtrack a little bit here. What would you say are the benefits and drawbacks of developing software in an open source environment, from the standpoint of the developer?
Weinstock: There are different ways of looking at that. Why would I want to participate in the development or why would I want to put myself more out there for free...
Place: I'll tell you at least one person's motivation for the latter. For the last two years, he's been unable to further his software at all, so since it was previously freely available, he said, "Okay, let's make this an open source project in the official open source project way. We'll get people who have ideas and have some suggestions for developing this further, and/or who have bug fixes to be able to maintain this thing."
Hissam: So, would you say that the rate of change on this project has increased or decreased, and have those changes been dramatic?
Place: Well, it's certainly increased. There's also one place where you can get an official source version that has the bug fixes in it, which you couldn't do previously.
Thomas: Would you say that putting out a program with open source code is a way of testing the market for it?
Pickel: Exactly. That's another good point that I wanted to make. One of the interesting things about open source is that you build on other people's software. When you release something, you never quite know how other people are going to make use of it. You learn quickly that way because they give you immediate feedback and contribute changes. It's a great way to figure out market demand.
Hissam: That would be a benefit. But if that evolution is unchecked, you're going to get the tcsh phenomenon. It's almost like a cancer: cancerous features.
Pickel: You choose the branch of the code that most suits your goals at a given time.
Weinstock: But that presents the consumer with a real problem, right? Which branch? What happens to the uneducated consumer who doesn’t have a basis for picking a branch?
Pickel: They go to places like Red Hat.
Place: If you want a version of BSD [a popular version of UNIX; BSD stands for Berkeley Software Distribution], which one do you pick right now? There are three versions of BSD that are all based upon 4.4, BSD Light, which was the last release. So which one do you choose?
Weinstock: Getting back to what you said about consumers going to Red Hat because they don't know how to make that choice: That's fine for an open source project where there is a Red Hat, but my guess is that most of them don't have a Red Hat. I mean, how do I know which Emacs to choose, for instance?
Plakosh: The only reason you have companies like Red Hat out there is because the distribution package for Linux is so large and so complicated—or at least it's viewed that way by the consumer. For a small piece of software, you're not going to have these distributors.
Pickel: Going back to your point about what to do if there is no Red Hat: I think these companies are out there for the purpose of infiltrating the corporate world—getting this kind of software into the corporate world. The techies and the geeks aren't necessarily interested in a Red Hat, though they may use it because they don't necessarily care to package all the software. But there are projects that don't have corporate backing behind them, or a very organized way of going about things. They're just not going to make it to as large an audience. They won't make it into the corporate world quite as easily.
Hissam: Every techie and geek on the planet right now, and every open source activity, started with dreams of IPOs [initial public offerings of stock]. They want to be the next millionaire. They want to start the next company.
Pickel: If you look at the past year or so, that might be one of the motivations behind open source software: people think they're going to make a killing off it. If you look over the past 20 years, there haven’t necessarily been financial reasons. One of the ways you get paid for leading a successful open source project is by getting your name out there, by getting well known, by becoming the guy who started that project.
Is Open Source Right for the Department of Defense?
Weinstock: Let’s talk about open source as applied to our Department of Defense client. What are the advantages of developing something using the open source model? When applied to the DoD, the notoriety factor is probably not important to them.
Place: There's another question that I should like to raise with respect to DoD customers. What systems do you envision the DoD would like to build with open source? Is it the weapons systems? Is it the payroll systems? How many people out there are interested in the payroll system?
Weinstock: It would seem to me that we're probably talking about subsystems first of all. Pieces of systems.
Place: So we’re talking back at the level of things like the operating system (OS), the database, the underlying components—the bits that we take for granted. That's one of the issues, when we talk about DoD customers. We need to understand that they're not going to build systems with this stuff.
Weinstock: But they will build systems that contain parts.
Place: Then the question is, which parts? Clearly it's going to be exactly those things—the OS parts, the database parts, the GUI [graphical user interface] parts.
What Are the Criteria for Success in Open Source Development?
Hissam: We can go back to the premise that these large organizations, be they DoD or not, think that they can get something done with access to a large, talented pool of engineers. There's some belief that they can get access to a lot of peer reviews, beta testers, people out there to look at their software and make it better and get it done quicker. That seems to be the running belief. If that's a model of success in Linux, then it must be true for every piece of open source software. But we should debunk that. Past performance should not be used as an indicator of future performance. That's one of the reasons that the Software Engineering Institute has to start looking at the processes that are used in open source development. What are the criteria that have to be there in order for it to be successful?
Past performance should not be used as an indicator of future performance.
Place: There are instances of projects that have been very successful. I would claim that Linux certainly has been one of them. It has achieved a level of reliability. The BSDs are reasonably successful, and some other open source things are reasonably successful. One of the common themes I've seen through either open or freely available source projects over the last 20 years is that there has been a substantial body of software—i.e., the system is basically there—before its release. People are bug fixing rather than developing new features, so that you've got a system with a structure and a design, and other people are now fixing the things that "he forgot" or that "he got wrong."
Weinstock: That suggests that it should start off with a user base or some sort of base of people who care about it.
Place: You certainly need people to care about it, and the people who care about this typically are the users.
The user base of Linux was people who developed software, not users of software, per se.
Plakosh: But if you look at how Linux kicked off, it kicked off by being more or less the toy of software engineers to build upon. It came with a lot of people's research projects. There were a lot of people looking at this functionality and that functionality. The user base of Linux was people who developed software, not users of software, per se.
Place: That’s a good point. The other thing, in thinking about what has been successful, is that the initial release was something that was a well-designed system.
What Are the Advantages to the User?
Thomas: Let's talk a little bit about open source from the user's perspective. What are the advantages to using open source software?
Hissam: Let me start off by cutting to the chase. It's a two-edged sword. The advantages are that users can get the latest and greatest and the fastest fixes. The disadvantages are that they have to get the latest and greatest and the fastest fixes. They might spend 75% of their time using a product and 25% of their time upgrading the product.
Just because a product is out in open source doesn't mean that I, as a user, have to track it.
Plakosh: That's not necessarily true. Just because a product is out in open source doesn't mean that I, as a user, have to track it. There are a lot of internal releases that I don't need to worry about or that I may not want to worry about. That perspective is somewhat from the mentality of the world that we live in, developing software.
What Are the Security Implications of Using Open Source Software?
Pickel: From a security perspective, you could look at it from a couple of standpoints. You really have to pay close attention to the software because if the community becomes aware of a vulnerability, then they're going to exploit it. So you need to beat them.
When you were talking about the double-edged sword, I realized that it also applies to the perspective of the developers in that there's an advantage to having people working on your software, but you also have to be ready to deal with them. If you have a lot of demand, and a lot of people developing your software, it's going to be tough to deal with them.
If the community becomes aware of a vulnerability, then they're going to exploit it.
Place: You raised the issue of security. What trust do you place in open source software, given that it's changing so rapidly? How much analysis can you do on the 5,000 fixes that came in last week?
Weinstock: Do you use Linux in a secure environment?
Pickel: Absolutely. Actually, I run Linux on all my machines. I'm not going to look at every single line of code. I'm not going to look at every update. But the thing is that there are people out there who are.
Weinstock: You hope. Do you believe that every nook and cranny of Linux has been looked at with that in mind?
Pickel: Not necessarily. But I believe that a lot more nooks and crannies have been looked at than are looked at in closed source environments.
Plakosh: That's an interesting statement because I would tend to bet that there is a difference between theory and practice. In theory, you would think that you're releasing source and you would have people combing over the code looking for security holes and trying to fix them. In practice, I don't think that's necessarily the case. In theory, it sounds great: the more people have it, the more people are looking at the source code and the more people are going to try to find bugs or security holes so that they can try to fix them. That sounds great. In practice, I think the only people who are trying to do that are maybe people who are trying to break into a system.
Pickel: Exactly. And they're part of the community. If they identify a hole and start exploiting it, people will notice that.
Hissam: You're saying that even the bad guys, in a sense...
Pickel: ...they're part of your development.
Place: But you only find out after the fact.
Pickel: That's still a better environment than closed source.
Plakosh: Actually, that's not necessarily much better because some of these holes that you can find in open source code you never would have found if it was closed source. They never would have existed.
Hissam: If I were a hacker, I could get the latest distribution from Red Hat, go to my garage, close the doors, get a lot of Twinkies and Coke, and start mulling over it until I can find a hack. Then I can just turn on my modem and go attack somebody who's using Linux.
Pickel: Certainly, there's some potential of administrators not noticing. This is an issue that we deal with every day in the CERT® Coordination Center. It's quite common that somebody will break into a site and the site administrators don't know how it happened. But when you deal with the sophisticated administrator, you can usually track down what program was the source, the problem. Then, maybe there's a new vulnerability in that. So, a lot of times, we look through the vulnerabilities, get in contact with vendors, and have the problem fixed. So there, in that situation, a few people were compromised. But the community as a whole is now operating on more secure software.
Place: Then there is the difficulty of getting customers to upgrade with the patch. Don't underestimate the number of people who are behind the curve.
Thomas: Is it any better in a closed source situation?
Hissam: Closed or open source, when a vulnerability is discovered, people have to react. I think the other thing that is interesting is that a hacker can become very intimate with some of the underlying protocols that are used. There may be that somewhere in the code it says, "You'd better check for null value here and the packet header for this because if you don't you could lock up the machine." The hacker says, "Wow! I hadn't thought of that before. If I tried this against Windows, I wonder what it would do?" Just by reading the source code, they could learn some very sophisticated, obscure attack.
What Comparisons Can Be Made Between Open Source and COTS?
Weinstock: There's also a big push to use COTS [commercial off-the-shelf] software, and widely using COTS raises all sorts of problems. At least some of the same arguments that apply to COTS apply to open source software.
Place: You might get an instability argument more so with open source than with COTS.
Plakosh: You'll get quality arguments too.
Weinstock: But it's the argument that "you own the solution if you try to modify it in any way."
Hissam: Let’s say a contractor is using open source in a government program and they're having some success. Then they run into a roadblock. They decide, "All I have to do is change this one line of code and we'll save the government millions of dollars." And they do it. Now let’s say the open source community doesn't want to adopt that change because it's very specific to whatever the government is doing. Now the government—by virtue of a contractor—is in the business of maintaining and competing with that open source.
Plakosh: That may or may not be true. I think one of the best advantages to me as a software developer is to take somebody else's open source and save development time, if it does do that for me. And I don’t intend to track it in the future.
Weinstock: Yes, but suppose there's a security compromise that's found in some future version and the four-star says, "That rolls back to the version you modified seven years ago—or six months ago." Now you've got to find someone who's even smart enough to put the changes in.
Plakosh: No you don’t. You've taken over maintenance of that. That’s fine, and you move on.
Weinstock: And the technician who worked on it seven years ago is still there, on the payroll, and there ready to help?
Plakosh: No. But we’re just talking about another method of software reuse. You're equating it to a product and having to track versions, rather than someone looking at open source software and saying, "Man, I could use these features. Let me rip them out and use them." You’ve taken the code from an open source product and reused it elsewhere.
Weinstock: But you’ve lost a supposed benefit of open source, which is that vast community of developers.
Plakosh: But if you weren’t interested in that, it doesn’t matter. You’ve gained. You didn't have to write that. You didn't lose anything. That's the benefit to a lot of people who are using open source. They don't have to reinvent the wheel. Somebody's already invented it. Yeah, I have to maintain it. Yeah, I'd better take a look at what I'm getting. Yeah, I'd better check the quality of it.
Weinstock: I'm not disagreeing with you. That's certainly a valid, good thing about open source. But if the whole world had that view of open source, there wouldn't be open source. All I'm saying is that it's sort of outside the spirit of open source as I see it.
Plakosh: What's the spirit of open source? Open source has two motives. One is that you want other people to work on your code; you want resources. So you want to obtain free resources.
Weinstock: Right. And you taking the code and going you own way and not feeding back to the community does not accomplish that.
Plakosh: But that's one motive. It’s for your own—how can I put it—personal glory, corporate gain, whatever. The second advantage is for other people just to advance technology and to advance people's growth. People give things out so that some other people can learn how to do something. For other people, it fosters new ideas. That's why I give out source code. I don't do it for my personal gain. I do it so that other people can look at it and use it for whatever they want to use it for.
Weinstock: That's from a developer's perspective.
Plakosh: I think it's great if someone reuses and finds benefit from something that I wrote, and if it saves them some time. Just like if I'm going to write a piece of software, I go out looking. I'm not going to try writing everything from scratch when I know that there is software that I can lift.
Pickel: The real benefit of open source, in my opinion, is that you build on things that are already available. You're furthering technology, and then people build on what you've done.
Place: You get the function you want as well. That's the other thing. If you're buying from Chuck's House of Software, you get what Chuck wants to sell you, not what you want.
Pickel: But you do have to be a developer to get that. I've been a user/developer of these things for a long time. If you're not a developer, you don't get it. I suspect that one of the results of this is that there will be more people who are developers out there. It's going to convince more people to look at the source code.
About the Participants
Scott Hissam is a member of the technical staff in the COTS-Based Systems Initiative at the SEI. He has 14 years of software development experience in industry, defense, and research. Hissam's principal areas of expertise include distributed applications, secure software and systems engineering, networking protocols, and operating system internals. Much of his recent experience has been rehosting legacy database applications on the World Wide Web. Prior to the SEI, Hissam was with Lockheed Martin, Bell Atlantic, and the U.S. Department of Defense.
Jed Pickel is a member of the technical staff at the CERT® Coordination Center (CERT/CC) at the SEI. Pickel is a member of the CERT/CC Incident Response Team. Pickel earned a Bachelor of Science in Electrical Engineering from the University of California at San Diego and is pursuing an MS from the Information Networking Institute at Carnegie Mellon University.
Pat Place is a member of the technical staff in the COTS-Based Systems Initiative at the SEI. Recently he has been part of the development of COTS Usage Risk Evaluation (CURE), a method for early identification of COTS-based risk in system acquisitions. Prior work at the SEI has included investigations into communications in distributed real-time systems as well as system specification using formal techniques. He has also developed tools to support Web development at the SEI. Before joining the SEI, he worked at various companies either developing tools or formal specifications, sometimes both. He has been an adjunct lecturer at South Bank University in the Electrical Engineering Department and also at Imperial College in the Computer Science Department.
Dan Plakosh is a member of the technical staff at the SEI. He is investigating technologies, techniques, and methodologies for developing distributed systems using commercially available distributed object technologies. He is currently researching topics associated with component architectures such as the development of custom component frameworks. Prior to joining the SEI, Plakosh was the lead software engineer for the Systems Engineering Department (F31) at the Naval Surface Warfare Center. Plakosh has 14 years of software development experience in defense, research, and industry. Plakosh's principal areas of expertise include real-time distributed systems, network communications and protocols, systems engineering, real-time 2D and 3D graphics, and Unix OS internals. Much of Plakosh's recent experience has been redesigning legacy-distributed systems to use the latest distributed communication technologies.
Bill Thomas is the editor in chief of news@sei. He is a senior writer/editor and a member of the SEI’s technical staff on the Technical Communication team, where his duties include primary responsibility for the documentation of the SEI’s Networked Systems Survivability Program. His previous career includes seven years as director of publications for Carnegie Mellon University’s graduate business school. He has also worked as an account manager for a public relations agency, where he wrote product literature and technical documentation for Eastman Kodak’s Business Imaging Systems Division. Earlier in his career he spent six years as a business writer for newspapers and magazines. He holds a bachelor of science degree in journalism from Ohio University and a master of design degree from Carnegie Mellon University.
Chuck Weinstock is a member of the technical staff in the COTS-Based Systems and Dependable Systems Upgrade Initiatives at the SEI. Weinstock is currently involved with helping the U.S. Coast Guard to understand COTS-related issues in the procurement of a new fleet of cutters. He is also working with Hill Air Force Base in a study of the use of model-based verification techniques in the software development process. He and Scott Hissam are embarking on an effort to better understand the open-source phenomenon. Before coming to the SEI, Weinstock was on the staff of Tartan where he worked on optimizing compiler technology. Before that, he was with SRI International's Computer Science Laboratory where he worked on the Software-Implemented Fault-Tolerant (SIFT) computer system.