State of the Practice of Intrusion Detection Technologies
3 What Are the Significant Gaps and Promising Future Directions?It remains to be seen whether or not intrusion detection technology can live up to the promise of accurately identifying attacks. Many claims are made, but few have been proven in practice. The current generation of commercial ID systems uses a limited set of techniques to detect attacks. Attackers are rapidly improving their abilities to perform successful penetration, including developing ways to defeat ID systems themselves. We see a significant number of gaps between the current state of commercial ID systems and a future state in which they provide effective defense against attackers. This section explores some of these gaps and suggests some future directions for both research and development.
First we look at some external drivers that show why alternate approaches to intrusion detection need to be pursued. Then we address how greater network complexity increases the difficulty of detecting intrusions. For example, the insecurity inherent in current network designs and topologies makes networks more vulnerable to attack. Human and organizational factors¾two topics given insufficient attention to date¾are addressed next. For example, the current emphasis on automated diagnosis is at the expense of any human-computer collaboration. Another factor is that of product selection in a highly volatile and uncertain market. We then identify a number of areas where the underlying technology needs to be improved. Examples include reducing the frequency of false alarms and early sensing of attacks. There are several issues associated with data analysis that need to be addressed, notably audit data reduction and the collection of forensic data to support law enforcement. Finally, we provide an overview of some of the research that is being performed to address current weaknesses. Many of these issues are identified in other documents such as "Intrusion Detection and Response" by personnel from Lawrence Livermore and Sandia National Laboratories [S14].
Although the issues in this section show that intrusion detection systems need many improvements, current ID technology, used appropriately, can satisfy many of an organization's security requirements. Section 4.4 provides guidelines to assure the most effective use of the technology.
3.1 The Need for Alternative Approaches
3.1.1 Sophistication of Attack Strategies
Gap: Attackers continue to improve their ability to penetrate networked systems and ID systems cannot keep up.
Attackers are becoming more sophisticated in their use of automated tools, in their ability to remain undetected, and in their application of unexpected strategies. Also, the frequency of these attacks is increasing. In "Cautionary Tales: Stealth Coordinated Attack HOWTO" [B80], Dragos Ruiu writes "...two years ago, we got mapped and port-scanned for vulnerabilities once a month. One year ago the scan frequency was up to once a week, and these days we get scanned several times a day with real attack attempts at least once a week." The article also states that "...the technical level of the attacks is increasing at an alarming pace, and I haven't seen any documentation of these new attack techniques." Similar observations have been made in the article "Techniques Adopted by `System Crackers' when Attempting To Break into Corporate or Sensitive Private Networks," written by consultants with Network Security Solutions, Ltd. [B106].
The paper cited above by Dragos Ruiu [B80] identifies seven different phases of an attack, provides insights into the strategies used in each phase, and describes personal experiences defending against these attacks. The phases Ruiu identifies are reconnaissance, vulnerability identification, penetration, control, embedding, data extraction and attack relay, summarized below.
Reconnaissance: Reconnaissance entails the attacker probing the region around which a protected network operates, such as an ISP or a web site. In this phase, the attacker's purpose may be to determine the addresses of trusted hosts, to disable a trusted but less secure host, or to attack the ISP's router system. Such peripheral attacks can be given a low profile by sending the extracted information to several attacker sites and doing so at a low rate. It may be difficult for a network IDS to identify these activities since they occur outside the network's domain.
Vulnerability identification: Vulnerability identification involves discovering weak links through which the network can be accessed [B108-1]. With the automated tools available today (e.g., nmap [B129]), unauthorized scanning for vulnerabilities is very common. However, given the right detection software, the system administrator can often spot these. Web, mail, and DNS servers are examples of targets that are often exploited since there is considerable pressure to minimize down-time on the popular services provided by these targets and since these targets tend to be more exposed as they sit in front of firewalls. Vulnerability in the mail server may be particularly worrisome since the information transmitted may be confidential.
Penetration: Penetration implies defeating any security boundary perimeter such as a firewall. It can be accomplished in a number of ways. For example, the ability to execute scripts inside application programs can allow a system to be hijacked by delivering malicious code inside innocuous packages (e.g., through email). Such code can open tunnels that allow penetration through the firewall. Alternatively, a "mole" program might be installed surreptitiously by an insider that allows the same tunnel functionality. These attacks can be made to look like traffic associated with HTTP or FTP, particularly if the content is encrypted.
Control: Once an attacker has access to the network, the focus is on gaining control and removing signs of entry [B108-3]. These steps can consist of installing a small script that contacts an already compromised machine from which a larger program (e.g. Back Orifice) is downloaded. After performing these operations, a cleanup of log files (e.g., system, event files, file integrity checker files, and ID systems files) is performed by rewinding these files to their pre-attack positions. As Ruiu says [B80], "you can completely remotely control a machine and run programs on it, upload and download data, without any indication to the user other than occasional sporadic slowness¾which on Windows is almost indistinguishable from normal performance, and Linux and NT aren't much better."
Embedding: In this phase, the attacker ensures that he can still retain control, even in the event that he is discovered. Since gaining control as described above puts the attacker in an exposed position, the aim here is to assure that access from this point forward does not generate obvious symptoms. This can be accomplished, for example, by overwriting little-used system files with malicious code that can survive a system reboot. Ruiu describes an attacker who inserted a piece of code that erased itself if not contacted within a specified period of time. Another insidious attack involved inserting code into an EEPROM of a network card¾thus even when the operating system was reinstalled the exploit was not removed.
Data extraction: Surreptitious retrieval of information suggests that data be encrypted and trickled out at a low rate. An additional strategy is to hide outgoing data using a common protocol such as HTTP for cover, perhaps disguising the data as a JPEG or GIF file.
Attack relay: The final goal of an intruder may be to use the compromised network as a springboard for attacking other systems. Since leaving a history of attack activity from one site makes the intruder vulnerable, it is in his interest to have multiple attack relay sites. These provide the opportunity to distribute an attack, thus reducing the likelihood of being detected.
As can be seen, the spectrum of techniques available to an experienced intruder is formidable, and these are severely challenging the capabilities of today's ID systems. Improvements in detection capability are likely to be countered by new, equally creative threats.
To stay ahead of these, comprehensive measures, including more sophisticated and adaptive technology, fusion of multiple data sources, integrated human-computer diagnosis, and effective security policies and training, are needed. These and other issues are discussed in the following paragraphs.
Gap: Tools to support
attackers continue to improve, and are an increasing challenge to intrusion detection technology.
The intrusion detection business is growing rapidly. A major driver for this is the release of software (including operating systems) that have not been adequately tested for security. In addition, rapid increases in connectivity and data sharing leave operating systems more open to exploitation [B65]. Tools currently used by attackers offer significantly more power to less skilled individuals, greatly enlarging the user base for more sophisticated attacks. In addition, automated tools allow an attacker to penetrate and retreat quickly or to pursue slow scans which fall below most IDS detection thresholds. These factors make detection much more challenging. Scanning and remote management tools exemplify the problem.
Scanning tools allow the attacker to determine system characteristics. These tools can recognize the version of the operating system that the host is using. This enables an attacker to exploit vulnerabilities specific to the configuration. The most widely known of these tools is SATAN [B66].
Recent tools such as sscan [B99] and nmap [B119], [B59] are widely used at the
present time. The latest version of nmap allows identification
of over 100 operating system release versions and can
camouflage its attack by sending out decoy packets that appear
to come from multiple innocent sites so as to hide the
intrusion source. Nmap makes it much easier to survey a
network slowly, at a rate that does not alert security systems
[B61], [B569-b2].
Port scanning allows an attacker to identify what services a host supports or does not support. Port scanning usually exploits the TCP, UDP or ICMP protocols to elicit responses from a probed host in order to identify active ports. Sscan also identifies a system's vulnerabilities. It allows one to write customized scripts that can automatically be executed to exploit the vulnerabilities that it has identified. Scanning tools such as nmap and sscan can be used just as easily to assess the security of one's own system as they can to attack another systems.
Remote management tools allow a systems administrator to remotely access and interact with servers. Significant damage can be done if such a tool is subverted. Back Orifice [B64] is a good example of a remote management tool that is commonly used as an attack vehicle. It can be installed in unsuspecting Window 95/98 computers and used to control the system from a remote client version of the tool. Originally introduced in 1998, it has been revised as Back Orifice 2000 [B64-b2] and is now being promoted as a legitimate (and free) remote management tool. However, given its notoriety, and the fact that the distribution disk was infected by the CIH virus [B86], its utility is questionable. A competitor to Back Orifice 2000 is Netbus. It too is now released as a legitimate remote management tool, but suffers from the same legacy problems as Back Orifice. These tools can be detected by virus detectors. When in use, Back Orifice and Netbus produce network activity that should allow intrusion detection tools to sense their presence.
It is generally claimed that network ID systems have an advantage in that they are passive and therefore undetectable by intruders. However, L0pht Heavy Industries is promoting a tool called AntiSniff [C23] which "came about due to the need to determine which machines on a local network are in promiscuous mode (i.e., collecting and examining data that is not destined to them)." The stated motive for developing this tool is the detection of illegally installed sniffing software. However, it could probably detect the operation of a network IDS. It is possible that this technology could be used by an intruder to sense, and consequently avoid, a network IDS running in promiscuous mode.
New vulnerabilities will inevitably emerge, and intruders will build tools that automate their exploitation. Tools used by attackers can provide capabilities that are difficult to mimic manually (e.g., decoy and rapid-action attacks), but these attacks should be detectable using ID systems. While intrusion detection has parallels in the virus domain, intrusion detection is more complex and the field is immature. Even so, the competitiveness (and possible survival) of an IDS vendor may very well depend on the rapidity with which that vendor can provide updates that effectively defend against the latest attacks.
Gap: There is an increasing challenge resulting from the rapidly growing numbers and changing goals of intruders.
Computer intrusions have been occurring since at least the
1960s. However, with industry and government becoming
increasingly dependent on networks for doing business,
computer intrusions with the goals of obtaining
economic/competitive advantage, political/military
intelligence, and financial gain have become more prevalent.
Andy Briney and Barbara Rose state in the paper "Study
Confirms Increased Security Risks of E-Commerce" that "...
companies conducting business online are 57 percent more
likely to experience a proprietary information leak and 24
percent more likely to experience a hacking-related breach.
Overall, the number of companies hit by an unauthorized access
(hacking/cracking) breach increased nearly 92 percent from
1997 to 1998" [S33-5].
Figure 3.1, taken from the
InformationWeek Global Security Survey [S30-2], summarizes the
responses of 2700 security professionals to the question "Who
do you suspect as the source of breaches or espionage in the
past year?" There are several interesting items. First, the
percent of respondents who suspect the group "computer hackers
and terrorists" of committing security breaches or espionage
has dramatically increased (up from about 14 percent in 1998
to 48 percent in 1999). This group has replaced "authorized
users and employees," the dominant concern in 1998. This
information is consistent with a conclusion reached by Harry DeMaio, president of Deloitte
& Touche Security Services [S33-1]: "The internal threat has been the
highest threat for so many years that it's almost a knee-jerk
reaction at this stage, but as more and more remote users gain
increased access to the system, the origin of the threat is
changing." With the increased use of outsourcing, the
rise in intrusions from contract service providers has
dramatically increased (from 9 percent to 31 percent of
respondents). The figure shows a significant concern in 1999
about the activities of "foreign governments," although no
1998 data was available. Interestingly, the threat from
competitors is not perceived as high (about 3 percent of
respondents), customers being rated a greater concern (16
percent of respondents).
As stated in the InformationWeek
Survey [S30-2], "Overall, security problems are becoming more serious. A year ago, half of the companies surveyed said they suffered no system downtime as a result of security breaches. This year, only 36% could make that claim." As seen in
Figure 3.2, viruses appear to be by far the largest (and growing) concern, while unauthorized network entry, information loss, data integrity loss, and denial of service all compete for second place (between 10 and 13 percent of respondents). Ninety-five percent of responding organizations indicated that they use virus detection software, while about 78 percent indicated that they had installed a firewall. A surprisingly high number of respondents (48 percent) indicated they had installed an intrusion detection system. However, since data was not available in 1998, the growth rate cannot be accurately estimated.
These figures reveal the concerns and organizational drivers in computer and network security. Overall there appears to be a greater awareness for the need for security and a shifting belief as to who the intruders are (e.g., external attackers relative to insiders). There is also a strong concern about viruses (64 percent reported virus infections), and virus protection has become ubiquitous. The rush to electronic commerce (and its financial implications) is opening up pathways for exploitation that were previously nonexistent. This issue alone may serve as a significant driver in motivating e-companies to install ID systems.
Gap: Encrypted messages can contain malicious information that cannot be detected by ID systems.
Message encryption is a problem, especially for network-based intrusion systems. Encryption makes the practice of looking for particular patterns in packet bodies futile. Useful analysis can be performed only after the message has been decrypted on the target host, and this often occurs within a specific application. Driven by commercial and defense needs, message encryption is likely to substantially increase in the near future. For this reason alone, greater emphasis needs to be placed on host-based or application-based ID systems that have the ability to view message content even if the message is encrypted in transit.
While encryption may make intrusion detection more challenging, this is likely to be offset by its positive benefits. With effective encryption, theft of information becomes much harder, and the motivation to penetrate computer networks is significantly diminished. IPSEC provides a mechanism (encapsulating security payload) that can be used to hide both the contents and addresses of network packets between cooperating agents such as firewalls [B96]. However, this renders the actual source, destination, and contents of the packet opaque while they are in transit between agents. If an IDS is positioned along the agent-to-agent path, it will be unable to determine the real origin or destination of the traffic. IPSEC implementations for IPv4 exist and its functionality is part of the IPv6 [B96] standard.
Like commerce using credit cards, organizations conducting commerce through the Internet will probably come to expect and accept a certain level of abuse as the cost of doing business. The same is likely to be the case with the use (or lack thereof) of encryption. One advantage of public-key encryption is that it may reduce the amount of information that ID systems have to collect and analyze by allowing them to ignore authenticated messages.
Gap: Certain types of attacks defeat the ability of ID systems to detect intrusions.
Intrusion detection systems will come under increasing threat from attacks since they represent the front-line of defense against attack. The paper "Insertion, Evasion and Denial of Service: Eluding Network Intrusion Detection" by Ptacek and Newsham [B26-b] was the first widely distributed paper to examine IDS vulnerabilities. The paper states that "...first, there is insufficient information available in packets read off the wire to correctly reconstruct what is occurring inside complex protocol transactions and next, that ID systems are inherently vulnerable to denial of service attacks." With respect to the first problem the issue is "...that a passive network monitor cannot accurately predict whether a given machine on the network is even going to see a packet, let alone process it in the expected manner." With respect to the second point, a denial of service attack results in the IDS failing open. This implies that the IDS ceases to operate but does not prevent the rest of the system from operating or communicating with the outside world. Thus, an attacker who can shut down an IDS can then operate with relative impunity within the rest of the network.
Insertion attacks result from the IDS accepting packets that the target host
rejects. In this way when the IDS reconstitutes a message from component
packets, the message contains the information in the spurious packets and
these make the message appear benign. However, if the target host rejects the
spurious packets, the malicious string is reconstructed. Conversely, in
evasion attacks, the IDS rejects packets that the target host accepts. Again,
the messages that the IDS and target host see are different, and thus the
attack message may get through. Unless the IDS can be set up to see exactly
what is on the IP stack of the target host (a non-trivial problem), this
problem will persist, and argues for the need for diverse and possibly
redundant means to identify intrusive patterns. This issue is discussed
further later in the context of data fusion (Section
3.6.4).
Ptacek and Newsham point out that there are two primary areas in which IDS behavior is likely to differ from host behavior. The first has to do with the handling of errors. ID systems tend to ignore checksum errors, inconsistent flags, and other phenomena that cause a packet to be dropped by a host-based protocol stack. This allows an attack to be disguised by inserting packets that will be accepted by the IDS, but dropped by the host. While it might seem simple to ensure that ID systems faithfully mimic host behavior, this is not done. The reasons stem from a variety of causes, ranging from performance considerations to ignorance of the subtle aspects of Internet protocols on the part of people who implement ID systems. The protocols themselves are ambiguous, leaving some details open to multiple interpretations so that different host operating systems can process the same packet stream with different results and still conform to the protocol. For example, packet fragmentation and reassembly combine with retransmission to create a complex situation that is handled differently in different operating systems. Unless the IDS knows how each operating system that it is protecting deals with pathologies involving fragmentation, etc., it is possible for the IDS to see an attack where none is present or to miss an attack that will succeed on a host.
Already attackers are using the above information to develop programs that will defeat intrusion detection systems. The article "Defeating Sniffers and Intrusions Detection Systems," in Phrack Magazine [B35] describes a variety of approaches to implement insertion and evasion attacks, and provides code to do so. The report "Bro: A System for Detecting Network Intruders in Real-Time" by Vern Paxson [R31] describes strategies for subverting intrusion detection systems. The paper classifies attacks into overload, crash and subterfuge.
The first two of these are variants of the denial of service attack, while subterfuge is related to insertion and evasion attacks.
The issues raised by Ptacek and Newsham are fundamental to the
way in which network ID systems operate and are not easy to
overcome. Insertion and evasion attacks require knowledge of
the host for which the message is intended, and network-based
ID systems do not have this knowledge [B89]. Because of the large
number of hosts on a typical network, and because of the
diverse applications being run on these hosts, installing
host-based protection will be a major challenge. However, the
alternative is the possibility that, even with an installed
network-based IDS, undetected attacks will occur. We are thus
likely to see increased efforts to integrate host- and
network-based systems, with a focus on minimizing the
installation, operating, and maintenance overhead. With such
systems there is a need to integrate data from diverse
sources.
This will challenge both data transfer limits and data
correlation (see Section
3.6.4).
Gap: Intruder access using undocumented modems increases
vulnerability.
Modems attached to networked machines can be points of vulnerability that escape notice [R79]. Attackers equipped with war dialers can identify and exploit such modems as points of entry into the network and can thus bypass the network's firewall and network-based IDS. In addition, security scanners do not generally include modem detection in their test suites. Because host-based ID systems analyze activity on specific machines, they are more likely to detect the presence of dial-in intruders on such machines, but only if they are configured to do so. In organizations with policies that forbid modems attached to workstations, this may be overlooked or subverted by a user who connects one.
If direct dial modems are essential, one can monitor the telephone trunk lines coming into the organization and record which lines have modem activity on them. The system can be designed so that all connectivity to the telephone network is through a single device. This allows call accounting on the device to be used for intrusion detection. Optionally, one can combine logs recorded at all interfaces between the computer network and the telephone network, using appropriate logging devices.
A solution to this problem is to make sure that no direct dial-in modems are attached to any networked machines, and that all communications go through the controlled points of entry such as firewalls or dial-back modems. Although the removal of all direct-dial modems is conceptually easy to implement, awareness of the issue needs to be raised. Thus the issue can be viewed more as a policy implementation gap than a technical gap.
Gap: There is inadequate protection against malicious mobile code.
Increasingly, executable code is being transmitted through
email attachments and the use of Java applets and ActiveX
objects. Given the added functionality and convenience that these capabilities provide, it is no wonder that they are popular. While ActiveX objects are less secure that Java applets, ActiveX's lack of security is reflective of its greater functionality, and the latter may win out. David Stang from Quarterdeck's Antivirus Research Center believes it is "...people's desire for style over substance. Ease of use and cool features will win over security and ill-defined security threats to privacy any day of the week" [B114]. These temptations come with considerable risk. ActiveX has a minimal security model and relies on what it calls "Authenticode"¾a mechanism for verifying the code's author. This provides little guarantee of safety since "...attackers can get certificates too" [R78]. The vulnerability of ActiveX was demonstrated by the Exploder object [B113], which, on being downloaded, "benignly" shuts down a machine. Other ActiveX objects may not be so benign, for example, an ActiveX object that transfers money from one account to another without authorization [B114]. Java, which supposedly executes in an isolated environment on a target machine, is not immune as it too may have vulnerabilities [R78],
[B114].
Some firewalls can prevent the importation of ActiveX and Java code, through the recognition of MIME types. However, current functionality provides no ability to discriminate between legitimate and malicious code and therefore one has to either accept all ActiveX or Java code or none. As indicated in "Why Monitoring Mobile Code is Harder than It Sounds [R78], code from JavaScript (a separate language from Java) can be made to generate Java applet tags on the fly. Firewalls can identify Java code by filtering on these tags; this mechanism allows Java to be surreptitiously introduced. Some efforts are being made to develop more "intelligent" firewalls that examine the behavior of, for example, Java applets.
The examination and monitoring of mobile code and its interaction with its surroundings appears to be a good conceptual fit with the functionalities of ID systems. These capabilities go beyond what traditional firewalls provide as firewalls focus primarily on perimeter defense. The tasks that such an IDS could perform might include
Given the increasing magnitude of this problem and the fact that the use of mobile code is likely to explode, we believe the link between mobile code and intrusion detection warrants further exploration.
3.2.1 Network Size and Complexity (Scalability)
Gap: ID systems cannot yet function across environments with diverse technologies and policies.
A network domain can contain a heterogeneous collection of
thousands of machines. This is challenging from a security
perspective. Compounding the challenge is the fact that
multiple domains may have to coordinate with each other on
security matters. This complexity will inevitably increase in
the future. To address the issue, several research efforts
have begun to explore the scalability of ID architectures.
Systems such as EMERALD [R2-a], , [R2-d], Hummer [R52-a], AAFID [R54-3], and NetSTAT [R30] all have such a focus.
EMERALD and NetSTAT have already been discussed in Section 2.1.1,
while AAFID and Hummer are reviewed in Appendix D.
Primary issues that intrusion detection faces in large heterogeneous systems are
If the above research vehicles are any guide, the predominant means of addressing such issues will be through hierarchical architectures. This allows for separation of concerns, such as different policies, different intrusion detection components, and different data repositories. However, network topologies way not always be hierarchical in form, and hence more general representations may sometimes be required.
Current commercial ID systems typically consist of a set of
sensor units that communicate with a central manager/analyzer unit [C10,
[C13],
[C15-a].
While these systems provide central data processing, they do
not intelligently correlate data from multiple sources (see
the discussion of data fusion in Section 3.6.4), and thus lack an
integrated systems overview of intrusive activity. As networks
become larger and intruders become more sophisticated, this
type of data correlation will become increasingly necessary.
There is hope that these problems may be addressed somewhat as
a result of standards activities in the ID community that are
attempting to address the issues of a common data exchange
format [B42-1] and the
application programmer interfaces for ID systems [R9]. (See Section 3.2.3 for details.) These
efforts are to be encouraged. However, given the dynamic
nature of ID systems and the highly competitive market, it
remains to be seen if they will be successful.
Gap: Operating systems are not designed to operate securely.
One reason that intrusion detection systems are in demand is that operating
environments are not secure. In fact, it could be argued that the demand for
openness encourages lax security. Peter Loscocco, et al. [B65-a], [B65-b] state that because of issues such
as rapidly increasing connectivity, application sharing, and use of mobile
code, operating systems must place increased emphasis on security. Operating
systems must be able to more effectively support policies related to access
control, authentication, and encryption. While more secure operating systems
have been built, these are not in the mainstream. However, it is argued that
"...the threat posed by the modern computing environment cannot be addressed
without the support from secure operating systems."
To meet the goals of a secure operating system, Loscocco [B65] describes several mechanisms. First, the paper describes a mandatory security policy "...that is considered to be any security policy where the definition of the policy logic and the assignment of security attributes is controlled by a system security policy administrator." Secondly, it discusses the concept of a trusted path¾that is "...a mechanism by which a user may directly interact with the trusted software, which can only be activated either by the user or the trusted software and may not be initiated by any other software." Supporting the trusted path is the concept of a protected path, i.e., "...a mechanism that guarantees a mutually authenticated channel to ensure that the critical system functions are not being spoofed." The need for the operating system to support access control and encryption capabilities is discussed.
The paper suggests a variety of "gains" from the use of a secure operating system, among them being that it protects against tampering with secured applications, permits safe execution of untrustworthy applications, and limits the scope of potential damage due to penetration of applications. Through policy, a secure operating system would more tightly constrain legitimate users from accessing unauthorized data and would prevent malicious applications from accessing data in ways that violate the intent of their user. Given the more robust security characteristics described above, the role of an ID system would be more focused and more manageable. False alarms would likely be reduced, and the need to perform under stress would be reduced.
DARPA has identified a range of "research challenges in operating system security" [R94], one of which is the meshing of prevention and detection strategies. In this paper the author suggests that
Because these are currently lacking in today's operating systems, the paper concludes that collected audit data "...is vulnerable to tampering or spoofing, and this leads to lessened faith in intrusion detection analysis."
There is a long history of secure OS development, largely funded by the DoD. Because it failed to achieve commercial acceptance, this work is generally seen as a failure. Given the competitive pressures to deliver a product's next version, security is not given high priority in design or testing. In addition, the need for products' openness and ease of use makes it increasingly difficult to incorporate security features that don't restrict the user. However, with the rapidly growing use of e-commerce, security is becoming a more critical issue, and the future demand for operating systems that are secure at a more fundamental level and provide a secure basis for intrusion detection may very well force changes.
Gap: ID tools are unable to interact, making correlation of results difficult.
Both commercial and research tools tend to be monolithic, consisting of integrated sensing, analysis, and management components. Such interfaces among components as may exist are closed and proprietary. As a consequence, existing tools cannot directly interoperate, and correlating data is difficult. This prevents all but the crudest attempts to increase the confidence in alarms by looking at data from multiple sensors, to integrate data from disparate locations in a system (such as will happen with a distributed attack), or to correlate information from different ID systems. The fact that some ID systems are able to communicate with network management systems such as HP OpenView and Tivoli via SNMP does not alleviate this problem. The issue is appropriate semantic interaction among systems rather than control of individual ID systems.
The lack of interoperability has been recognized by both the vendor and research communities each of which is addressing it in their own way. Under the prodding of DARPA, the research community is developing the Common Intrusion Detection Framework (CIDF). Within the Internet Engineering Task Force (IETF) is the Intrusion Detection Working Group (IDWG). Although vendors were invited to participate in the CIDF effort, most chose not to do so. On the other hand, the IDWG is primarily a vendor oriented group, though it appears not to be a typical IETF working group.
CIDF is developing the Common Intrusion Specification Language (CISL). This seems to be a fairly heavyweight attempt to provide a basis covering all conceivable ID architectures, but comments at a recent DARPA Principle Investigators' meeting indicate that several research groups have found a need to develop their own specification languages which may or may not be extensions of CISL. DARPA has established a working group to discuss the issue of intrusion specification languages, but it is not clear whether this group will make a significant contribution in the area as, to date, activity has been minimal. There appears to be little interest in the CIDF approach among IDS vendors, and it is unlikely that this effort will have any near term impact on the commercial world.
The IDWG is primarily a vendor group operating within the IETF framework. Currently, it has a requirements document for intrusion detection message exchange and proposals for protocols, a data model, and data representations within the framework established by the requirements document. If the normal IETF process is followed, these proposals will be accepted by the group, and trial implementations will be produced by one or more participants. Once interoperability among multiple versions is demonstrated, the proposals can proceed towards standardization as one or more IETF request for comments (RFCs). This will probably require a year or more to occur.
The IDWG approach is much more modest than that proposed by CIDF. Presently, only alert data is proposed for interchange and the amount of information required to be transmitted is minimal. Appendix E contains further information about these two efforts.
Gap: Network-based ID systems may not see all traffic.
Network intrusion detection systems analyze the traffic passing though network segments in an attempt to detect attacks on both the network infrastructure and on the attached hosts. In the former case, classes of attacks that would be difficult to see from the perspective of a single host can be seen easily at the network level. In the latter case, the network IDS must make sure that its view of what a given host will see is consistent with that of the host. There are a number of factors that make both situations problematic.
Recent years have seen a trend towards switched, as opposed to broadcast, local network segments. This trend means that there may not be any location within an enterprise from which both internal and external traffic is visible. Careful analysis of the network configuration is required to determine whether or not a network IDS is capable of seeing all the traffic of interest. This trend is further exacerbated by the growing use of virtual private networks and encrypted tunnels which render network traffic opaque to the IDS. Even SSL Web connections can provide cover for some forms of intrusions. In addition, the bandwidth of both internal and external network connections is rising rapidly. Multi-megabit connections to the outside world are becoming commonplace. One hundred megabit internal Ethernet connections are the rule with gigabit segments offered by most router and switch vendors. Network-based ID systems are less and less able to keep up with these traffic loads. Router vendors whose backplanes must pass these traffic levels are unable to allocate the computational capacity necessary to perform stateful examinations of this traffic. At traffic levels that permit analysis, many intrusions require the accumulation of substantial state information for their identification. Examples include the creation of probe or scan histories, tracking of incomplete protocol exchanges, such as TCP opens, reassembly of fragmented packets, etc. This opens the IDS to denial of service attacks based on exhaustion of its internal resources. These trends will most likely continue for the foreseeable future. As a consequence, network-based ID systems will have decreasing applicability and will have to be replaced by alternative approaches.
When a network IDS attempts to infer the effect of traffic directed towards a specific host on that host, additional problems arise. Depending on the position of the IDS in the network, it may be impossible to ensure that the traffic seen by the IDS and the hosts that it is protecting are the same. If there are multiple paths between a host and the outside, only some of which pass by the IDS, it is possible that the vagaries of Internet routing may send some packets between a given external source and the target host through routes that are not visible to the IDS. A coordinated attack from multiple hosts may make this even more likely. Even if the IDS sees all traffic that is delivered to the host, there is always a possibility that the host will drop packets due to resource limitations within its own protocol stack. It is not feasible for the IDS to model the behavior of all hosts on the network with sufficient accuracy to avoid problems of this nature.
3.3.1 Cooperation between Organizations that Lack Complete Trust
Gap: Mechanisms are lacking that allow competing organizations to cooperate for their common good.
Independent organizations may need to cooperate with each other on computer intrusion issues. For example, organizations could be members of a competitive community that, out of enlightened self-interest, need to cooperate on common security issues. In some cases it may be necessary to transfer potentially untrustworthy information between such organizations or to query or notify one another about intrusion-related incidents. Alternatively, there may be a need to share vulnerability-related information within an application domain.
A recent paper, "Research Issues in Cooperative Intrusion Detection between Multiple Domains" by Frinke, et al. [R66], investigates data sharing between entities that have different security policies and incomplete trust. The work focuses on the issues of authentication (Who sent the message?), reliability (Is the data accurate?), and integrity (Is the received message what was sent?) of intrusion-related information. The tool, Hummingbird, which has been used to explore these issues, is reviewed in Appendix D. This issue of untrustworthy communication is briefly alluded to in the SRI paper "Future Directions for Intrusion Detection" [R59].
Research at Columbia University has been addressing the issue of sharing
models of fraudulent transactions within the financial community [R53]. Clearly, financial data can be highly
proprietary, and there is a great reluctance within that community to admit to
being a victim of financial fraud. However, in order for the financial
community to defend against fraud, they must cooperate and share some
security-related information. The approach relies on cooperating financial
organizations developing local fraud detection models (using a
pattern-directed inference approach), and then combining these models across
organizations in order to generalize the detection capability. This approach
allows an abstraction of the financial data such that proprietary information
is removed while still retaining the ability to share essential
intrusion-related information. The research vehicle for this work is JAM (Java
Agents for Meta-Learning; see Appendix
D).
Broad coordinated attacks against computer networks of organizations that share similar characteristics and roles (e.g., financial institutions, military agencies, utilities) will become more frequent. To combat these attacks, mechanisms such as those described above are helpful in supporting coordination and cooperation among potential victims. While there is a clear advantage to such cooperation, many organizations (especially in the financial industry) are very sensitive about making public statements on issues pertaining to their security. Recent proposals for a nation-wide intrusion detection network (FIDNET) put forth by the Government Services Administration immediately ran into a storm of criticism over such issues. The success of these efforts is therefore likely to depend more on each organization's ability to cooperate with its peers than with any technical issues.
Gap: Human skills are not adequately used to support diagnosis of intrusions and to determine resulting actions.
People excel in detecting patterns, particularly if the basis
for these patterns is not fully understood (e.g., people have
hunches, a sixth sense, a feeling in the gut). We have strong
skills in identifying unexpected relationships between
entities, and use analogy to recognize new relationships.
Symbolic reasoning and graphical thinking are central to these
abilities (see Section 3.6.6). On
the other hand, computers are able to search through, compute
on, organize, and graphically display massive quantities of
data without introducing the types of errors that people are
prone to make. These capabilities complement the human
skills.
Since the detection and diagnosis of an intrusion can be challenging, it may take diverse points of view to reach a reliable conclusion. Combining the complementary analytic strengths of a person and a computer to reach reliable conclusions thus makes sense. To date, however, this symbiosis has been marginally exploited [C4]. Commercial network-based ID systems indicate that an intrusion has been detected and provide only low level data to support conclusions. They offer no information to help determine if a conclusion is credible with respect to the evidence or diagnosis.
There is little opportunity to examine the reasoning behind
the conclusion or to interact with the system in order to
examine alternatives. Many security managers will not want to
interact with an IDS in this manner¾they would rather be
handed the result¾but this can only lead to a false sense
of security. However, with many types of intrusions, the
probability of a correct machine diagnosis may be low, and
operator involvement becomes increasingly necessary if a
correct outcome is desired. The issue of correct diagnosis is
critical early in an attack scenario (see Section 3.4.1) when subtle
information may only hint at a problem. In this case, human
insight and experience may be crucial to assure a positive
outcome.
Of particular interest is Rasmussen's model of operator mental activity [B71]. This model suggests that operators of complex equipment structure their cognitive reasoning into three domains related to skill-based behavior (when human response is relatively automatic), ruled-based behavior (when rules, known through the operator's past experience, are applicable), and knowledge-based behavior (when there is sufficient uncertainty that human reasoning from first principles must be performed). These three regimes directly map to the regimes in which an IDS can automatically determine the nature of an event, when there is some standard interactions between the operator and the IDS, and when the IDS must rely significantly on the operator to diagnose the problem. The latter case directly relates to the detection of a new type of intrusion.
There is a considerable body of research on human-computer
interaction [B71], [B72], [B73], and this discipline
appears to have significant, though unrecognized, relevance to
the technology of intrusion detection. We believe that this
fruitful research area needs to be aggressively
explored.
Gap: ID systems are unable to identify attackers and their goals.
In criminal cases (particularly serial murders), profiling perpetrators based on evidence is an accepted technique. Analogously, the evidence left by an intruder on a computer network provides evidence, both as to who the perpetrator is and (perhaps) what his goals are. The tools and techniques an attacker uses, and the manner in which they are used (e.g., frequency of use of certain commands or command sequences), provide a fingerprint that is probably unique to an individual. While profiling legitimate users has not met with much success, it is not intended that this type of intruder profiling be automated. Rather it is a computer-supported but manual task that attempts to identify the idiosyncrasies of the attacker's behavior.
Trying to directly identify an attacker's goals could be difficult since a direct inference between actions and goals is likely to be weak. For example, an attacker's goals might include either curiosity or information theft. Curiosity may be a mild irritant. However, the consequences of information theft may be the loss of a crucial competitive advantage. Based on observing only the symptoms of the intruder's activity, it may not be possible to distinguish between the two. Nevertheless, in other cases, where the physical outcomes are different, some weak inferences could probably be made between activities and goals.
Given the ability to identify an attacker from his prior methods of operation, and given an understanding of the attacker's prior goals, it may be possible to infer where his current activity is leading. For example, an intruder who, in previous attacks, was after military secrets will probably continue in this vein, while an intruder who was vandalizing an ex-employer's database will probably continue along this line. Insights into the attacker's goals would be invaluable since they provide a basis for action. Actions might include continued passive observation, attempting to trace the attack's point of origin, blocking the intruder's progress, locking him out of the network, or providing a honeypot for entrapment [B108-4].
Attacker profiling for intrusion detection has not been
explored to any great extent [R80], [R81], but might have
promise. Much historical information on intrusive activity has
been gathered within security support organizations such as
CERT/CC. This data could be mined to determine whether
intruder profiling provides an accurate characterization of
individual attackers. Additional information may be found in
documented case histories [B19],
[B27],
[B28],
[B76],
[S19]. Note that this area of investigation
may provide significant support for forensic analysis (see
Section 3.5.3).
Gap: Widely accepted ID terminology and conceptual structures are lacking.
The nomenclature used by researchers and practitioners in the field varies widely within the community. Indeed, this variation in terminology has motivated the authors of this report to develop a set of definitions for technical terms used within the report (Appendix A). SANS has developed a set of terms for use by the ID community [B6].
There have been a variety of attempts to organize the field's conceptual structures, especially taxonomies defining types of intrusions and intrusion detection techniques. Some of these are identified in Appendix D. Of particular importance is the need for a taxonomy of vulnerabilities, as this would provide a more rigorous basis on which to develop intrusion detection methods. A simple categorization of vulnerabilities by Kragen Sitaker can be found in "How to Find Security Holes" [B100], while a paper from Matt Bishop [R82] discusses vulnerability analysis as a precursor to classifying vulnerabilities. The latter paper describes his initial efforts to construct a classification. In addition, there is a recent effort, hosted by MITRE, to sidestep the taxonomy and develop a comprehensive listing of exploitable system flaws known as CVE1 [R83]. The rationale for this effort is given in a paper [R83-b] that appeared near the beginning of the CVE effort.
There are several efforts that attempt to define shareable
schemas, universal primitives and robust taxonomies for
vulnerability information. While we remain interested in and
supportive of these efforts, we note that there appears to be
no consensus on these issues yet. Our problem is that we need
to achieve interoperability among our tools right now.
Moreover, the problem preventing us from achieving this
integration is much more basic than the definition of good
schemas, primitives or taxonomies. The problem is that there
is no consistency in the community with regards to
identifying the vulnerabilities.
To be fair, this is characteristic of an immature and developing field as standards, especially for terminology and acceptable practices, are typically codifications of widely accepted practices.
Within such a young, dynamic field as intrusion detection, it may be some time before concepts and their relationships to each other become stable. However, if an effective taxonomy of vulnerabilities could be developed, it would provide a more solid basis for reasoning about intrusion from a defender's point of view, and would provide a more rigorous foundation for building intrusion detection systems.
Gap: The volatility in the ID marketplace makes the decision about which product to purchase and maintain difficult.
The commercial intrusion detection field is rapidly changing with respect to
This bewildering sea of change makes it very difficult for the buyer of products to make rational purchasing decisions. During the course of our investigation, we have seen almost constant change. Because of the rapidity of this changing environment, we have had to rely on web material for a substantial portion of the information that appears in the report. In several cases, reports that appeared in one location early in the process were either moved to another location or disappeared before we finished. In at least one case, the company responsible for seminal work was acquired by another; reports were moved, then removed. Another product was spun out of its original home as a separate company which was subsequently acquired by a third company.
Several of the products that we attempted to deploy underwent revision while we were experimenting with them. Although we recommend a comparative evaluation of commercial products in the ID arena, some products will likely become obsolete during the process. Therefore, evaluation may need to be an ongoing process to be useful.
These changes are likely to continue and provide a further indication of the immaturity of the field. Because this process is largely unpredictable, it is difficult to give advice about how to deal with it. Although we see continuous change, the change does not appear to represent significant breakthroughs in either technology or usability. The best strategy may be to make purchasing and deployment decisions based on the products and information available at decision time and not to look back or to allow second guessing of the decision. At the same time, users should realize that change is inevitable and that improvements are made, so that purchasing and deploying a new IDS periodically may be necessary.
3.4.1 Sensing an Attack Before Damage Occurs
Gap: ID systems are unable to identify many types of intrusion in their early stages.
The more lead-time an IDS can provide in identifying an attacker's activities, the better the chances that damage can be averted. Thus it makes sense for an IDS to know the early signs and sequences of activity that characterize an attack. For detecting signs that characterize an attack, statistical profiling has been shown to indicate hostile probing activity. For detecting sequences, work has been performed on developing temporal models of attack sequences. In addition, implementation of manual intrusion detection policies can be effective to support early warning. These approaches are described below.
Attackers who are not familiar with a network generally probe
the system to acquire information on the system's
characteristics (See Section
3.1.2). It is argued in "Towards Trapping Wily Intruders
in the Large" [R81] that
there are significant statistical differences between the use
of network services for legitimate purposes and the use of
network services to support intrusion. For example, while the
presence of TCP resets may or may not indicate hostile
probing, the characteristics of suspicious use are easy to
identify. While this technique does not detect many intrusive
behaviors, it could be a useful early warning
indicator.
A more general strategy is to model attack scenarios. In this
approach, there are notions of attack sequences that lead to
compromised configurations. As a hypothesized attack evolves,
evidence is built up for and against the probability of that
type of attack based on predefined attack models. This
approach was first proposed by Garvey and Lunt [R23] and later implemented
in the STAT methodology [R3], [R4], [R5], a state transition
approach. It allows the IDS to provide early warning of an
attack, with growing or diminishing confidence of the
existence of the attack as the evidence unfolds. (See the
review of NetStat in Section 2.1.1.2)
A similar approach uses colored Petri nets to capture the
attack sequences [R7]. (See
Section 3.6.5)
While tools can be of significant benefit in helping to detect
early signs of intrusion, enforcement of manual policies can greatly help this area and does not require major investments in technology. By frequent review of logs, processes, and other network/system information early activity can likely be spotted. The SEI reports Preparing to Detect Signs of Intrusion [B116] and Detecting Signs of Intrusion [B98] provide practical advice to system administrators on what to look for in this area. (See also
Section 3.4.3.)
More research needs to focus on identifying early indications and warnings of an attack. The more an intruder has penetrated a network the more difficult it is to assess the state of the system and to know how much damage has been done. With effective early warning, much of the need for complex tools to deal with eliminating the penetration and recovery from damage is removed. This is an area where human experience and insight could play an important part in guiding early, but probably unreliable, computer-based analysis.
Gap: ID systems are unable to take appropriate automatic action in the event of an attack.
Automated response to attack (i.e., aimed at recovery or defense, not reprisal) is appealing since
Some current commercial ID systems do support operator intervention. For example, CMDS supports four response levels of manual intervention: ignore the warning, increase observation, denial of access, and emergency shutdown [C13]. RealSecureTM allows an associated firewall to be reconfigured to reject traffic from a specific IP address [C15-a1]. These actions do require a human operator to be in the loop.
Given the current maturity of IDS technology, the dangers of automated response are significant, and outweigh the above advantages. With the frequency of false positives that exists in the current generation of ID systems, the potential for inappropriate response to misdiagnosis is too high. In addition, automated response could be exploited by a perpetrator whose aim is to induce denial of service by spoofing an attack from a legitimate user. Finally, as indicated in the article "Intrusion Detection and Response" [S14], the issue of limiting the effect of automated response so as to prevent cascade and livelock2 failures that may be caused by the response system needs to be addressed. While recognizing the dangers of automated response, the same article suggests that automated response is clearly necessary, especially in critical infrastructure elements involving speeds or volumes beyond the human capacity to respond. However, until the accuracy of ID diagnosis is significantly improved, appropriate automated response remains high risk.
There is a clear need for ID systems to be able to distinguish between scenarios that are absolutely guaranteed to indicate invalid usage and every other kind of scenario. There are a large number of these "absolutely invalid" scenarios. One additional way to define "absolutely invalid" is that the victim is willing to take programmed actions in response to their occurrence. If the victim is unwilling to take such actions, the scenario by definition is not "absolutely invalid."
Gap: ID systems provide little to no support for damage recovery.
After an intrusion has occurred and damage (e.g., data
corruption, data modification, information theft, or system compromise) has resulted, can intrusion detection systems help? Current ID systems provide little direct support for damage recovery [S14],
[B67]. Commercial network ID systems most often rely on simple string matching to detect known patterns of intrusion. This provides neither any indication of the extent of the damage nor how the damage could be repaired. Host-based intrusion detection systems such as Centrax [C12] may be better candidates for supporting system recovery.
One tool that currently does provides meaningful support for post attack recovery is Tripwire [C22]. This file integrity-checking tool was originally developed at Purdue University and later commercialized. The Tripwire Security Systems Web site states: "Tripwire monitors all servers and clients on a network, detecting and reporting any changes to critical system or data files. Tripwire can absolutely, unequivocally determine if a protected file has been altered in a way that violates the policy set by the administrator" [C22]. Tripwire creates cryptographic signatures for all critical system data and stores the signatures in a protected database, preferably off-line or on read-only medium. Upon administrator request, Tripwire recomputes the signatures and compares them with those in the database, detecting any changes called out in the Tripwire configuration file. If undesirable changes are detected, the system can be restored to its original configuration. A more limited response of this type is provided by Network Associate's CyberCop's tool. This provides a "AutoRestore" capability that can delete Web pages corrupted by an intruder and can replace these with the original pages.
As described in Section 3.3.2,
much can be accomplished through manual support of recovery
activities. The SEI report Responding to Intrusions [B123] provides information
in this area, particularly within the context of developing a
recovery policy. In addition, the article "Some Tips on
Network Forensics" by Ranum [B68] provides some practical
insights about intrusion response.
A controversial issue is that of reprisal against perpetrators. An IDS can provide information on where a detected attack is emanating from (or at least where the attack appears to be emanating from) so that "attack-back" could be performed [B44]. However, if the intent is to damage a perpetrator's capability (as opposed to sending a warning), the dangers are many. Given the problems of, for example, spoofed IP addresses and false diagnoses, retaliation is more likely than not to hit the wrong target. This is further complicated by the fact that wireless phones can make it much more difficult to track the ultimate source of the attack. In the event that the attacker is successfully identified, the victim better be sure that his defenses are secure and that the attacker does not have a backdoor to his system. Counter-reprisals could be devastating. Approaches to mitigating this problem through an improved ability to track perpetrators are discussed by Cohen in "Providing for Responsibility in a Global Information Infrastructure" [B69]. While the IPv6 standard [B88] provides some intriguing possibilities with regard to intrusion protection through authentication, it is unlikely that this protocol will deter a serious attacker.
Recovery can be significantly simplified if one has enforced
policies and procedures in place to perform frequent backups
(to a write-once device or removable medium) and has
mechanisms in place to minimize the likelihood that backups
are not corrupted (using, for example, an integrity checker).
However, with an experienced intruder, subtle modification of
files or theft of information may be difficult to detect.
Information theft may be especially challenging because all
evidence of the intrusion can be removed if audit trail
information is stored on an unprotected disk. In situations
where security is critical, it therefore makes sense to store
audit records on write-once media on a physically secured
device. Because these records can be voluminous, compression
of audit records may be necessary (see Section 3.5.2).
Gap 1: ID systems cannot keep up with normal traffic loads.
Gap 2: ID systems cannot detect and counteract intruders that are starving the ID systems of resources.
To be effective, a network ID system must be able to analyze all incoming packets. If the IDS cannot keep up with the throughput, then intrusive patterns may not be seen and a vulnerability is created. In evaluating ID systems, a NSTL survey states "...sure, ID systems spot attacks as advertised-on empty networks. They also work well on heavily utilized Ethernet segments. But fill up a fast Ethernet segment with traffic and that vigilance vanishes; in fact, no product detected all the attacks when the network was heavily loaded" [S20]. This comment is consistent with the concern raised in a CSI roundtable discussion [B15] which likewise indicates that "...most IDS products can't even keep up with 10Mbps Ethernet speeds. The networked environment is rapidly moving way beyond that speed." Thus an attacker may bypass an IDS out of "dumb luck" simply because of the current high traffic.
The premeditated use of resource exhaustion to attack or avoid an IDS is discussed in some detail by Ptacek and Newsham [B26-b]. Three types of resources that can be exhausted are identified: CPU cycles, memory, and network bandwidth. In the first case, Ptacek and Newsham state, "... the IDS spends CPU cycles reading packets, determining what they are, and matching them to some location in the saved network state... An attacker can determine what the most computationally expensive network processing operations are and force the IDS to spend all its time doing useless work." Regarding memory exhaustion, Ptacek and Newsham [B26-b] state, "...an attacker can create a stream of meaningless events and, by having them continually stored, eventually exhaust all disk space on the IDS which will then be unable to store real events." Finally, with respect to network bandwidth, they further state, "...an attacker can overload the network with meaningless information and prevent the IDS from keeping up with what's actually happening on the network." The underlying problem is that a network IDS must analyze all network traffic moving through it, while most other components only have to deal with subsets of this data. Thus, the IDS is more likely to be bottlenecked. In a sense, some types of ID systems make easy targets for resource exhaustion attacks because they devote resources to maintaining state information about partially completed attacks on multiple hosts. Unless the resources of the IDS have almost the same aggregate resources of the protected network, the ID system may be quite vulnerable. Thus, if the IDS is the only means of protection available, a successful attack on the IDS may leave the rest of the system vulnerable.
High growth in future network traffic will make the matter worse. To address this issue there is a need to
Gap: ID systems are not able to recognize new attack strategies.
ID systems that rely on matching patterns of behavior that represent signatures of known attacks are unable to detect previously unseen attacks with different signatures. ID systems need to protect against unknown or new attacks; the following approaches may be worth pursuing:
The first of these is discussed in Section 3.4.3 and provides an
approach that is based on existing technology. One weakness of
this approach is that the perpetrator must already have
compromised the network to the extent that files or system
parameters have been modified. Even if a backup of the
environment allows one to revert to a prior valid
configuration, sensitive information may be lost and/or
stolen.
Privileged programs (such as UNIX sendmail and fingerd) can be vulnerable to abuse through the execution of root-level commands that were not intentionally designed into the programs. Work by Ko, Fink, and Levitt [R96] has explored mechanisms to detect violations of the legally specified behaviors which, for the programs examined, are quite simple. Thus, any behavior which is outside the allowable set can be detected. This approach is as applicable for unknown exploits as it is for known exploits.
The third approach is the oldest¾namely anomaly detection
through user profiling. This approach was originally proposed
by James Anderson [B34] who hypothesized that "it is possible to characterize the use of a computer system by observing the various parameters available through audit trails, and to establish from these observations, `normal' ranges for the various values making up the characterizations." Statistical profiles of legitimate users are stored, and these are compared to the behavior of current users. If the current behavior deviates significantly from the behavior reflected in the stored profile, an anomaly is indicated. This approach was also pioneered in the early 1990's by SRI International [R1-a],
[R1-b],
[R1-c], and
[R1-d], and was incorporated into at least one commercial system [C11]. However, this approach has never seen widespread adoption since
The fourth approach to detecting unknown attacks is to
determine what represents normal behavior within networks,
hosts, or applications, and to flag activity that does not
reflect what is expected. This is an extension of user
profiling to protect other infrastructure assets. One approach
was originally inspired by biological immune system principles
that have a "sense of self" [R50]. It was hypothesized
that many of the attributes of the immune system (e.g.,
recognition of foreign bodies, self-repair, and no trusted
components) could be transferred to intrusion detection. In an
initial investigation [R48], short sequences of
system calls, resulting from the execution of normal
privileged processes, were stored in a database. These
represented the "self," and were compared with system calls
from process sequences that contained abnormal behavior. At
least for simple cases, it was shown that such abnormalities
could be detected. The notion of "self" has been extended to
look at distributed CORBATM-based networks [R60]. In this example, a
CORBA application is considered as the organism whose cells
are the clients within that application. The focus of the
research is to determine if "rogue" events can be detected
after training the system to recognize normal behavior, as
reflected by system call sequences. In both examples, the
results were encouraging, but the authors recognized that
additional work was necessary before these notions of "self"
could be translated and applied for intrusion detection.
Further details on "computer immunology" can be found in Section 3.6.1 and Appendix D. While encouraging,
this approach is still experimental.
If a method of attack is unknown, then only indirect evidence of its presence can be detected, and this makes the intrusion detection problem hard. The techniques described above attempt to infer novel intrusions, but, with the possible exception of file integrity checkers, success has only been on an experimental scale. Use of an immunological analogy is, in principle, very powerful. However, it remains to be seen if system call sequences (i.e., the entities that provide a notion of the computer's "self") provide a sufficiently unique discriminator of authorized and intrusive behavior. Underlying any technique that attempts to protect against unknown attack is the need to provide an IDS with a notion of "self," since anything that is "not self" is by definition anomalous. Unfortunately if the notion of self is not sufficiently accurate, too many anomalies will be detected raising the number of false positives to an unmanageable level.
Gap: ID systems provide little to no guidance for responding once an attack has been identified.
Current commercial ID systems provide little guidance about appropriate responses when attacks are detected. Because of the high frequency of false alarms, vendors may be reluctant to give advice. Multiple diagnoses can result from a set of symptoms, and it would be confusing if different guidance was given for each of these. However, there may be intrusions where the symptoms are unambiguous, and where the consequences of not acting are serious. In these cases, some guidance could be extremely valuable.
In the competitive world in which ID systems exist, there is a reluctance to divulge proprietary signatures that represent known patterns of attack. This includes the publication of information that reveals inaccuracies of signatures. With a library of openly available signatures, the problem would be reduced. The ID community would be free to scrutinize the logic of the signatures and to provide improvements in their precision (i.e., the consistency with which signatures would predictably identify known attack patterns). Precision would help ID users determine the best response and give them a high degree of confidence that it was appropriate.
Gap: The accuracy and adequacy of IDS signatures cannot be determined.
The proprietary nature of the signatures for most commercial ID systems makes a detailed discussion of their accuracy and adequacy difficult. For the most part, signatures seem to be at a very concrete level, not far removed from simple string matching. This may be adequate for very simple attacks such as those that can abuse a CGI-bin executable with a single command, but are probably inadequate for sophisticated, multi-stage attacks.
Among the research and open source tools, the situation is somewhat different. The STAT family [R4] models attacks using abstract state machines. The machines are constructed to recognize generalizations of known attack scenarios. If the individual constructing the recognizer has sufficient insight into the attack and uses abstraction to its maximum advantage, a recognizer for a class of attacks rather than for an instance of the class may result. This happened during the 1998 Lincoln Labs evaluation when NSTAT was able to recognize a new "user to root" attack that had not been present in the training data.
Given that many of the observed attacks are the result of scripts distributed by the attack's developer, and used essentially as is by numerous "copy cat" intruders, concrete patterns have a definite role in the intrusion detection arena, just as they do in the virus arena. Were it mature, the IDS vendor community would be rushing new signatures to their users as fast as new attacks are recognized. Our worry is that the attacker community and "script kiddies" that seem to be responsible for much of the attack traffic are not the community that we should be most concerned with. There are obvious variations on most scripts that would escape detection by all commercial (and many research) ID systems. A focused attacker, targeting a system for the purpose of extracting, inserting, or modifying information is likely to be able to escape detection long enough to accomplish their goals.
These issues have a bearing on how the vendor community needs to address the release of signatures. It appears that what will distinguish the more successful ID vendor is
In the latter case, this probably means more rigorous testing of signatures, improving the robustness of signatures to variants of the attack mechanisms (as discussed above), and minimizing the false alarm potential of the signature. Clearly there is a conflict in addressing the two goals of timeliness and quality, but the more effective vendors will develop processes that can balance both.
Gap: There is no generally agreed upon figure of merit that can be used to characterize IDS performance. Acceptable performance is similarly difficult to define.
The typical IDS looks at a series of events and tries to identify those that represent an intrusion. The events may be log records from one or more monitored services on a host, packets on a network, or any other surrogate for activity within the monitored domain. Normal activities as well as intrusions may be represented by single events or by combinations of events. At the service monitoring level on a host, an activity may give rise to a series of log records (e.g., an ftp session might generate log records indicating the start of the session, successful authentication, files transferred, directories examined, and termination of the session). These records may be interspersed with the records of other simultaneous ftp sessions as well as records from other services. A monitor on the network segment serving the host would see the packets that made the session, along with all the other packets that passed along the segment during the same period. In one case, an event is a log record, in the other, it is a single packet.
When an attack occurs, it may be represented by a single event
or by a sequence of related events. On the network it may be
possible to accomplish the attack with a single packet, but it
may also be possible to fragment that packet into a number of
small pieces as discussed further in Section 3.1.5. It may require a
number of related packets to carry out an attack, and again,
it may be possible to fragment many of these. Similarly, a
single log record may serve as a positive indication that an
attack has taken place or it may require a sequence of such
records, extracted from the log stream to make this
determination.
We belabor this point because it is germane to the discussion of IDS accuracy and precision. From an operational standpoint, we are interested in having a system that identifies as many of the real attacks as possible while raising as few false alarms as possible. One way to characterize the former is in terms of the percentage of attacks identified by the system. Assuming that we can evaluate the system by subjecting it to known attacks, we ought to be able to compute the percentage of attacks identified. Unfortunately, for reasons that will be discussed below, even this is not easy. Ideally, we would like to subject the system to a large amount of activity that contains no attacks. The numerator of the false alarm rate is the number of attacks falsely identified; what is the denominator? If we associate an attack with an activity of some kind, we need to count activities, a problematic undertaking. Besides, there is no guarantee that the events that contributed to the false alarm all came from a single activity or even a related set of activities.
In the only set(s) of test data with which we are familiar, that used by Lincoln Labs to evaluate DARPA sponsored research systems [B29], the term "session" is used to identify related sets of packets. A session is characterized by a start time, duration, service, source and destination (IP address and port). In the test data, attacks are associated with sessions and the nature of the attack is indicated in the labeled training data. Using the session as the unit of analysis is less than satisfactory3 since it does not provide accurate results if
Assuming that we can choose an appropriate unit of analysis (dubious) there are four possible outcomes for each unit:
Each of these cases can be described using Bayesian or conditional probabilities. In general, we want the percentage of true positives to be high and the percentage of false positives to be low. Unfortunately, these figures alone are not sufficient to describe a system in meaningful terms since we do not know how frequent attacks are (i.e., the percentage of analysis units that contain an attack).
Unfortunately (for the purpose of the analysis), the percentage of units that represent attacks is more likely to be on the order of one in a 100,000 or so. Optimistically assuming a 100% true positive rate, we will identify every attack. Now assume that the false positive rate is 0.1%, an objective given by DARPA as a target for its research. This will result in a false alarm rate of about 100 per 100,000 and the staff will have to examine 101 analysis units to find the one true intrusion. Recent papers by Axelsson [R77] explore this in detail.
Perhaps the most meaningful figure of merit for a deployed IDS is the ratio of the true positive rate to the false positive rate. Some experimentation is probably required to determine the range of acceptable ratios, but it is unlikely that systems generating a preponderance of false alarms will be acceptable. If we consider 5 false alarms for every real alarm to be the limit of acceptability, a system that identifies one real attack in 100,000 units of analysis must raise false alarms in no more than 5 cases per 100,000 or 0.005%, a far more stringent requirement than that imposed by DARPA. As the attack frequency falls, the false alarm requirement becomes more stringent. Translating these requirements into more easily measured base quantities such as packets or audit log records requires an understanding of the detection process and how it forms its analysis units.
The Lincoln Lab's evaluation of DARPA research systems has stated that the acceptable limit on false alarms is about 100 per day, arguing that this is the maximum number of intrusions with which a single analyst can deal. Even if an analyst can sustain this rate of intrusion handling, we doubt that the analyst will be able to sustain the level of performance necessary to identify the single real attack per day that is hidden in the 100 or so false alarms predicted by a 0.1% false alarm rate.
In summary, both the Bayesian detection rate and the Bayesian false detection rate, along with the traffic level and the attack rate, are factors in determining whether the performance of an IDS is acceptable. If the number of false alarms per day is acceptable, but the attack rate is too low for reliable discrimination by system staff, it could be desirable to add additional (preferably benign) intrusions into the analysis stream to provide both analyst stimulation and system calibration.
Gap: The false alarm problem is exacerbated by the fact that legitimate traffic often contains the kinds of pathologies typically associated with attacks.
The more theoretical arguments of the previous section are
reinforced by Paxton [R84]
who believes that "...the whole concept of [network] intrusion
detection is doomed to failure. Attacks on the monitor, such
as overloading it with too much traffic or using software
faults to bring the monitor down can be defended against, but
that still leaves the problem of `crud' that looks like an
attack but isn't one." Some odd-looking but legitimate traffic
includes
Paxton believes that the in inability to discriminate between "crud" and malicious packets will be exploited by attackers. For these reasons he feels that host-based ID systems are likely to predominate.
The resulting discussion leads to the dismal conclusion that false alarm rates in current network-based systems may be so high that system administrators will ignore all warnings.
3.5.1 Testing ID Systems
Gap: Very little has been done to objectively evaluate and test ID systems.
Section 3.4.8 briefly discussed
research systems testing. The misconceptions about commercial
systems are even worse. Reading vendor literature gives the
impression that commercial products currently provide robust
defense against intruders. However, independent IDS
evaluations (including the evaluations performed for this
study) indicate that this is not the case. Generally vendors
neither provide information about the performance of their
products, nor do they make their detection algorithms public.
The latter point is motivated by the need to hide attack
signatures from competitors, and the need to prevent intruders
from knowing exactly how the signatures work. A third, but not
readily admitted, motive may be that by publicizing
signatures, it will become apparent how simplistic many
signatures really are. Full public scrutiny of attack
signatures may, in the short term, result in an increased
number of intrusions, but in the long term the scrutiny would
motivate the development of more robust approaches to
intrusion detection. In the meantime, black-box testing is the
only public means to determine the diagnostic accuracy of
these tools.
Testing is difficult since an IDS cannot realistically be
examined in isolation but needs a network, or a network simulator, with which to interact. Several such environments are being developed within the research community [B7],
[B9],
[B16],
[B38] and we hope they
will migrate into use within the larger user community. Testing also requires significant volumes of complex data (either captured from a real network or developed synthetically) against which the IDS can be evaluated [R25,
[R92],
[R93]. In addition, many intrusions depend on one operating system or even the specific release of an operating system. Thus test data may need to be finely tuned to the operating environment.
However, "...some of the problems that we discovered were so basic (the conditions leading to these problems occur frequently even in normal traffic) that it appears as if no in-depth testing has been done at all", and "the most important issue that vendors need to address is testing" [B26-b].
Independent tests have been performed on commercial products (see Appendix D for details). However, given the immaturity of the field, the rapid evolution of existing products, and the frequent introduction of new products, these tests are often obsolete as soon as they appear. New attacks and variants of old attacks occur frequently, and test data can soon become outdated. Any useful testing methodology must address the issue of obsolescence. It must also address the general testing issues discussed above.
Penetration testing can determine whether an IDS system is functioning properly, as well as evaluating system or network security. This is performed by making "friendly" attacks against the network and hosts (and their ID systems) as opposed to testing an IDS prior to installation. While more reactive, this type of testing does not require one to develop large databases of test data. Human actors serve in the role of attackers, directly using the tools and techniques that attackers use. Penetration testing is discussed in the papers "Improving the Security of Your Site by Breaking into It" [B107], and "Broadening the Scope of Penetration Testing Techniques" [B105]. While penetration testing is an important assessment tool, it may only tell you that the IDS you have purchased is inadequate.
The testing of ID systems is difficult. Given the spurious results that most commercial ID systems too often generate, it is understandable why vendors are reluctant to be involved in public testing. Several organizations, mostly research oriented, are developing environments for evaluating IDS tools, but as these environments are themselves immature, they have not seen significant use in evaluating commercial products. Over the long term, IDS testing environments need to generate test data on the fly to address the growing number of new attacks.
Gap: Tools to effectively manage audit data are lacking.
For complex networks, storing raw real-time audit data for retrospective ID analysis may require impractically large storage capacities. Thus, there is a need to develop techniques that allow compression of the data so that loss of information, important to subsequent ID analysis, is minimized. Issues that need to be addressed in this area are
With the ever-increasing size and complexity of networks,
managing and storing audit data will become an increasingly
challenging but important problem¾not only to the IDS
community, but to network management in general (e.g., for
performance analysis). While data compression and filtering
technology currently exists, its a
Figure 3-1: Security professionals views on intruder
threat origins, adapted from an InformationWeek Survey [S30-2]
(Note that multiple responses are allowed, so that totals exceed 100 percent.)
Figures in this file are displayed in a separate browser
window. This window will remain open to display figures in this file,
although it might be hidden behind other browser windows.
Figure 3-2: Responses from security professionals on
security concerns, adapted from an InformationWeek Survey [S30-1] (Note that multiple responses are allowed, so that totals exceed 100 percent.)
Figures in this file are displayed in a separate browser
window. This window will remain open to display figures in this file,
although it might be hidden behind other browser windows.
3.4.5 Identifying Unknown Modes of Attack
We believe this should be a wake-up call to the research community. Research efforts in the intrusion detection field have focused increasingly on building ID frameworks that address higher level issues and abstractions. We believe that these efforts should, in great part, be redirected to address the fundamental issues of minimizing false alarms and quantitative testing of the resulting algorithms. If the false alarm issue is not resolved, all the frameworks in the world will not help.
The answer to this depends on the audit needs and is constrained by whether the IDS is network- or host-based. On the network side, is packet header information sufficient or is packet content important? If content is important, the storage problem is much more severe. Audit data storage needs could be reduced if the data is only sampled (this might also help with performance) but what is the basis for sampling? Since storage capacity is not infinite, there must also be mechanisms to discard or de-emphasized older data.
The most direct solution is to use data
compression techniques [B70]. This approach usually
allows the original data to be completely restored. However,
if loss of "unimportant" data can be accepted, then greater
compression can be achieved by filtering or using learning
techniques originating from the artificial intelligence
community. Techniques such as rule induction, neutral nets,
and genetic algorithms all provide means to compress data into
forms that require much less storage. These AI approaches tend
to generalize over that data, removing irregularities as they
do so. As a result of this data abstraction, they may also
provide insights into patterns in the data. However, full
restoration of the data is not possible, and this may have
implications for forensic analysis. These learning algorithms
are discussed in more detail in Section
3.6, Advanced Research.
Performance is affected by computational and communication overhead and there is likely to be a direct trade-off between these. Network and some host ID systems consist of a set of sensors and a central management unit. If the sensors transfer all the raw audit data to the central unit for storage, then the communication overhead will be high, and this may affect the overall performance of the network. Some of this overhead can, however, be reduced if the sensor units perform local data reduction¾with a resulting load on the local CPUs. Trade-off analysis, specific to the network, would have to be performed to determine the optimum balance between CPU and network traffic loads. This issue represents a "mini-gap." A different performance issue results from the use of learning algorithms. Given the large quantities of data that must be managed, such algorithms must be capable of incremental learning, but many are not.
Compression using learning techniques will generally require a degree of regularity in the data structures. This implies that, when raw data from different sources are involved (e.g., host vs. network data or data from two different types of IDS), the data will need to be integrated prior to compression. Integration may have performance implications and the practicality of integration will depend on the effectiveness of data exchange formats [R68-b]. The consistency and completeness of data from different sources may also be problematic. Finally, when integrating diverse data sources, there could be difficulty in determining the relative timing between logged events due to the use of different clocks.