Software Engineering Institute Carnegie Mellon

Organizational Models for Computer Security Incident Response Teams (CSIRTs)

[Abstract]   [Title Page]   [Preface]   [Acknowledgements]   [1 Introduction]  
[2 Establishing CSIRT Capabilities]  [3 Operational Issues]  
[4 Security Team--Using Existing IT Staff]  
[5 Internal Distributed CSIRT]  
[6 Internal Centralized CSIRT]  
[7 Combined Distributed and Centralized CSIRT]  
[8 Coordinating CSIRT]  
[9 Choosing the Right CSIRT Model for Your Organization]  
[10 Closing Remarks
  [Appendix Summary of Services Offered [Bibliography]   [PDF File]


8 Coordinating CSIRT

8.1 Overview

In this model the main focus of the CSIRT is to coordinate and facilitate incident and vulnerability handling activities across a broad, diverse, and usually external constituency. This coordination and facilitation can involve sharing information, providing mitigation strategies and recommendations for incident response and recovery, researching and analyzing trends and patterns of incident activity within the constituency, providing resources and references for incident management such as vulnerability databases, clearinghouses for security tools, or advisory and alert services.

There are different types of coordinating CSIRTs and each has a different level of authority in relationship to the supported constituency. One type of coordinating CSIRT may serve a specific constituency group--for example, a coordinating CSIRT for a multinational corporation. In this case the CSIRT may have authority to implement incident response solutions and mitigation strategies across the organization. However, it can be the case that the international pieces of the corporation are affiliate companies and not under the jurisdiction of the CSIRT.

Another type of coordinating CSIRT may serve a constituency made up of the various branches of a country's military. In this case, the CSIRT may have authority over all members of the constituency.

Another type of coordinating CSIRT may serve a whole country, province, or state. In this case the CSIRT will not necessarily have authority over the constituency. The same can be said for a CSIRT for a large or national research network, educational institution, or the general public. For example, the Internet community (which includes computer security experts as well as the general public) is the constituency for the CERT/CC. However, the CERT/CC has no authority over anyone within this constituency, but can affect change based on the value of the information and service provided to the constituency.

When the coordinating CSIRT has no authority, it can only act as an advisor to the constituency. It cannot make any decisions or take any actions on its own for specific systems that are affected. The coordinating CSIRT can provide high-level analysis and suggest recovery and mitigation strategies, but it is up to the constituency to decide to follow the recommendations. The coordinating CSIRT may be able, because of its position and reputation in the constituency, to influence the decision-makers to act for the overall good of the organization.1

Whatever the type of coordinating CSIRT, it always serves a distributed constituency. Usually the constituency consists of multiple, independent entities, however, they may be in similar sectors such as various military or financial organizations. These entities may even have their own internal CSIRT. In such cases, the coordinating CSIRT interacts with the internal CSIRT as a point of contact. Information and recommendations are passed on to the internal CSIRT, whose members then choose what to pass on to their own constituency.

8.2 Supported Constituencies

As already mentioned in the overview, this model concentrates on the coordination of many independent entities. Usually such entities are organizations that share some common characteristics that make them part of the team's constituency. Common characteristics that are usually found today are

Coordinating CSIRTs have a long tradition, starting in the early 1990s, of providing incident response services in multi-organizational constituencies (e.g., CERT/CC), and especially in the European research networks, the SURFnet Computer Security Incident Response Team (CERT-NL) and DFN-CERT are popular examples. While CERT-NL and DFN-CERT coordination efforts were focused on a particular bounded domain (a national research network), their informal constituency was much larger in practice. Being the only CSIRT available, at that time, in a specific country made them in reality the "default" coordinating CSIRT on a national scale. Although this posed some practical challenges to CERT-NL and DFN-CERT related to workload, charter, and authority, as time progressed and the development of other teams increased, the burden of being a default coordinating body for the unbounded constituency lessened. That being said, sometimes these CSIRTs still receive requests for assistance from these broader constituencies, even though there are other, more applicable CSIRTs that should be contacted. AusCERT, for example, although a membership-based CSIRT, still is contacted by other external groups who are seeking help in notifying sites in Australia concerning incident activity.

Today there are a number of CSIRTs that coordinate larger multi-organizational constituencies2 like the U.S. military, the U.S. federal government, various research networks, and to some degree the commercial entities that are peers within a single country. National CSIRTs, for example, will participate in coordination efforts across their constituency and probably with other national CSIRTs but concentrate their efforts locally at their constituency level in their day-to-day operations.

Some countries establish one coordinating CSIRT for a whole nation by providing government funding. An example of this would be SingCERT, which serves the Internet community in Singapore.

In countries where no other country-level coordinating CSIRT has been established, an existing CSIRT may extend its services to the bigger, informal constituency, making it in fact a national CSIRT. CERT-NASK, for example, became CERT-Polska early in 2001 and is now serving the Internet community in Poland.

Other coordinating CSIRTs may service a particular geographic region or sector. For example the Asia Pacific Computer Emergency Response Team (APCERT) works to coordinate CSIRT activity in the Asia Pacific area. The TERENA Task Force "CSIRT Coordination for Europe" (TF-CSIRT) does similar coordination work for the European Community.

8.3 Organizational Structure

As the coordinating CSIRT is most likely a dedicated team, it has a central location and manager.3 Ideally, the CSIRT comprises staff with expertise in all systems and platforms supported by the constituency. However, if the constituency is made up of many single, independent organizations, this is not usually possible. In that case, experts from the constituency or other trusted computer security organizations need to be identified to work with the team as needed. The CSIRT staff contains positions for triage and hotline handling, incident analysis, support, response, and coordination. The coordinating CSIRT may also have staff that perform vulnerability and artifact handling services. Administrative support staff is also required.

Although it is up to the coordinating CSIRT to determine what services to offer, the constituency can often influence what is provided based on their needs. Since a coordinating team in a large, geographically dispersed constituency cannot reasonably provide direct incident response on site, and since the coordinating CSIRT should not compete with the constituency's internal CSIRTs, the services generally provided will complement existing local services or provide value-added services not provided within the constituency. The main functions of the coordinating CSIRT are to act efficiently as the coordination center and to direct the response effort at various levels of the organizations that make up the constituency by providing advisories, alerts, training sessions, documented policies and procedures, and expert guidance. The coordinating CSIRT, acting as a neutral party, is able to synthesize information to form a high-level view of activity and then provide detailed analysis to those constituent members who do not have available resources or expertise.4 Many of these teams may need to rely on the coordinating CSIRT's analysis and guidance to determine appropriate response strategies.

8.4 Triage

In a coordinating CSIRT environment, the triage function is central to the operation of the team. It is a clearly defined point of contact. There are advertised descriptions of the services provided, hours of operation, and guidelines for how and what to report. Online reporting guidelines and online references are available to assist the constituency's staff in reporting and contacting the coordinating CSIRT.

Identified staff in the coordinating CSIRT perform the triage function. Explicit guidelines for what requests and reports are handled and what are not handled are developed and used by staff to assist in performing this service.

8.5 Available Services

The following sections describe services that might be provided in a coordinating CSIRT model. It is recognized that every team is different, so these are general descriptions based on observations of and discussions with other teams. The method in which the service is delivered assumes a certain level of infrastructure, staff, and equipment, which are discussed in further sections.

8.5.1 Core Services

Because of the structure and operational goals of a coordinating CSIRT, the following services tend to be the basic ones most often provided, although they are somewhat different from the normal core services discussed in Section 2.7.4.

Alerts and Warnings

Since the first CSIRT was created, this service has been part of the core set of services offered by coordinating CSIRTs. In the day-to-day operations, CSIRTs receive and triage all incoming information, especially concentrating on events that point to any risk the constituency might face. As part of the CSIRT work, they forward all information concerning alerts and warnings to the points of contact in their constituency. They also may create their own alerts and warnings based on information and research collected. Once information is distributed to identified points of contact, it is up to these points of contact to determine how much further this information is distributed within the constituency and to whom the information is disseminated.

After any distributed alert or warning, the coordinating CSIRT collects and evaluates feedback from the constituency to re-evaluate and further refine the assessment to better serve the constituency. As any feedback would be voluntary, this re-evaluation may be based on a low number of responses, rather than based on feedback from the whole constituency. The information could also be based on further research and analysis that the coordinating CSIRT performed itself.

Incident Analysis

The coordinating CSIRT undertakes analysis of incident reports received to determine the nature of the activity being reported, what intruder tool(s) were used, the scope of the activity, and the appropriate recovery or mitigation strategies to be applied. They are not usually reviewing incident artifacts and logs to recover a particular system, but to see what the basic attack strategy was, so they can correlate this information with other activity across the broader constituency. In-depth analysis or forensic analysis on affected systems would be done by the constituency's local CSIRT or security team.

Since a coordinating CSIRT most likely does not receive reports regarding every individual incident occurring in its constituency, it must make estimations of the scope and threat impact based on the reports it does receive.

The CSIRT performs incident analysis to understand what is occurring in the constituency. Based on its understanding of the overall picture, the CSIRT makes recommendations for strengthening overall security when possible. It is able to identify high-level intruder trends and attack methods, and use this information to provide suggested strategies for securing and defending constituent systems.

Incident Response Support

Because the coordinating CSIRT is not on site and not devoted to one specific constituency, its main focus will be to provide support to many constituency organizations, which could include other CSIRTs. This support can take various forms depending on the needs of the overall constituency. Supporting activities can include

Information can be disseminated via intranets or extranets, email, phone, or mailing lists. Each constituent entity determines who receives the information and assistance and who follows any distributed guidelines to perform the response operations and tasks.

Incident Response Coordination

In a coordinating CSIRT model, the incident response coordination service is one of the main services or functions of the team. With dedicated resources, the team can provide comprehensive tracking, recording, and dissemination of information for the constituency. By consolidating collected information, the team is better able to identify similar attacks, artifacts, exploits, trends, and patterns. Potential new threats to the constituency can also be identified and mitigation strategies developed and distributed. The coordination work done in this model is more a matter of information exchanging and facilitation of interactions between the parties involved in the recovery or analysis of the ongoing incident activity.

In this model, although it is desirable it is unlikely that the team will have expertise or familiarity with all platforms and operating systems inside the constituency. Therefore it will need to call upon external experts from constituency sites, vendor organizations, other computer security organizations or other CSIRTs to assist in the actual analysis. The coordinating CSIRT can act as a facilitator or a main point of contact for bringing these various organizations together. It can also be a main distribution point for disseminating the resulting response or mitigation strategies to the rest of the constituency.

Because the coordinating CSIRT is a well-known point of contact for its constituency, it may receive warnings and alerts from other organizations that need to be redistributed to the sites and constituents involved.

Vulnerability and Artifact Response Coordination

Similar to the way that the coordinating CSIRT provides incident response coordination, it can also be effective in providing vulnerability and artifact response coordination. These coordination functions are possible because of the wider variety of information the CSIRT is able to gather and analyze from its diverse constituencies and because the CSIRT has more time to devote to collecting and analyzing the information. This ability to collect and synthesize information that can be shared with the various components of the constituency is one of the greatest benefits of the coordinating CSIRT.

Another part of this coordination effort is to inform constituents about the results of various analyses of vulnerabilities and artifacts along with any remediation strategies.

Announcements

Because the CSIRT has access to information from the various organizations within its constituency and from other security experts and groups, it can present a broad picture of incident activity to the constituency. It can do this through general announcements based on this comprehensive information. These announcements are intended to raise the awareness of the constituency towards new trends and areas of concern for the security of the constituent organizations or of the constituency at large. The coordinating CSIRT also can provide information to help the constituency proactively defend its critical assets. This may take the form of letting constituents know of newly found vulnerabilities and artifacts, so they can check their systems and remove or fix the problems before they are exploited.

Technology Watch

This service is another that can be provided by the CSIRT to the constituency as a value-added service. The members of the coordinating CSIRT can focus more time on performing a technology watch function than most of their constituent organizations, due to their dedicated staff. This can be an extremely beneficial resource provided by the CSIRT.

Individuals on the coordinating CSIRT are assigned this function for the various supported technologies and platforms as resources are available. The information they collect is consolidated to highlight current attacks, threats, trends, and other relevant items. This synthesized information is made available to the rest of the CSIRT staff via a secured intranet or extranet and is then, in turn, used to further create value-added information for the constituency.

Security-related information that is of interest to the constituency can be posted to a mailing list or an Internet discussion site as a method of keeping network, system, and security administrators up to date. It can also be used to raise the level of security awareness for all members of the constituency. Such a site can provide educational benefits by allowing people to post questions that can be answered by the CSIRT staff if time permits.

Security-Related Information Dissemination

The coordinating CSIRT may be able to provide this type of service for its constituents who do not have time and resources to collect and disseminate this information.

To provide the constituency wide access to security-related information, the CSIRT can establish a centralized web site (and corresponding FTP site if necessary.). The coordinating CSIRT collects information on security trends, best practices, and tools, and provides this information to either its points of contact or to the whole constituency. If desired, recommended tools and software updates or patches can be made available to provide authenticated versions for reference in alerts and warnings.

As with the centralized CSIRT model, coordinating CSIRTs may provide translation services to distribute security information to the constituency in their native language.

Awareness Building and Education/Training

Most coordinating CSIRTs engage in some form of awareness building, education, or training for their constituency. This might involve developing training classes on security and incident response issues, tutorials on attack types and mediation strategies, or even research into incident and vulnerability trends. Because of this, we include these services in the core services list for coordinating CSIRTs.

Members of the coordinating CSIRT may be assigned to visit constituency site locations to provide briefings or security awareness training. CSIRT staff can also provide instruction on security issues, tools, and recovery techniques. Sometimes this is done as a for-fee service, and sometimes it is done as a free member service.

8.5.2 Additional Services

In addition to its core services, a coordinating CSIRT may choose to offer other services. The following services are those most likely to be provided.

Vulnerability and Artifact Analysis

A coordinating CSIRT may have the means, expertise, and time to analyze various vulnerabilities or artifacts that it receives through reports or through its own research, while its constituents may not have the time or the expertise to do this type of work. The CSIRT can focus on those vulnerabilities and artifacts that might have a potential impact on its constituency, or it may analyze vulnerabilities and artifacts to provide general public information rather than information specific to the constituency.

A good example of this can be seen in some of the work done by the CERT/CC. This coordinating CSIRT provides a knowledgebase of vulnerability information to the public. The CERT/CC has dedicated staff to analyze reported vulnerabilities and work with vendors to determine the status of a vulnerability in various products. This is not a service that many CSIRTs have the time or resources to perform. Many different constituencies can benefit from this work, without having to replicate this service at a local internal team level.

Vulnerability and Artifact Response

After completing the analysis of vulnerabilities or artifacts, any relevant information for mitigating or repairing a vulnerability or detecting and removing an artifact is passed on to the constituency. This information may be distributed as an alert, advisory, or even as a technical document. For example the CERT/CC provides vulnerability information and mitigation strategies via the Vulnerability Notes database and the Vulnerability Reports Catalog, both pieces of the CERT/CC Knowledgebase.5 In a similar manner, MITRE's Common Vulnerabilities and Exposures (CVE) database also provides information about and a catalog of vulnerabilities.

Usually this response effort is limited to the provision of information and mitigation strategies. However, some CSIRTs may offer additional for-fee services that involve traveling to a site to help actually repair and recover affected systems. Others may provide a fee-based service to help sites install patches.

Development of Security Tools

With the proper staff time and expertise, members of a coordinating CSIRT may become involved in developing security tools that may be used by members of their constituency or by other CSIRTs. For example, the CERT/CC has developed tools such as AirCERT (Automated Incident Reporting) and specialized secure mailing tools. JANET-CERT and DFN-CERT are involved in developing various incident tracking systems. Other teams may develop virus or IDS signatures or other tools, scripts, and patches for use in response activities.

Other Services

Generally a coordinating CSIRT does not provide services involving configuration and maintenance of security tools, applications, and infrastructures; security audits or assessments; or intrusion detection. However, in rare instances, these can be provided as for-fee services, if the team has the time and expertise to perform these functions.

8.5.3 Impact on Security Quality Management

In most cases the coordinating CSIRT does not work with other parts of the constituency to provide security quality management services. Instead it provides general guidelines that can be used by members of the constituency to improve the overall security of their enterprises. An exception to this is if the coordinating CSIRT was hired in a consulting or managed security service provider capacity to specifically perform these services. Another exception to this would be if the coordinating CSIRT was actually coordinating other internal CSIRTs within the same organization, such as a coordinating CSIRT in an educational institution that coordinates activity across other CSIRTs at branch campuses.

Services that the coordinating CSIRT might be hired to perform if they are external to the constituency or that the CSIRT might coordinate if they are an internal coordination center include providing security consulting and assisting with the development of security policies and business continuity plans.

Other services that the coordinating CSIRT might provide include the development and delivery of training courses, tutorials, and security awareness briefings. These have been included previously under "Core Services," as most coordination CSIRTs provide these services.

8.6 Resources

The following staffing, equipment, and infrastructure resources should be considered when implementing a combined CSIRT model.

8.6.1 Staff

A coordinating CSIRT provides a core staff that devotes 100% of their time to coordinating the incident handling activities of their constituents.

This staff contains the following individuals:

The size of the team will be determined by the size and diversity of the constituency. For additional tasks and functions that support the work of the core staff, arrangements need to be made in advance with

The coordinating CSIRT can also cooperate and collaborate with other security or organizational experts from within the constituency when specialized expertise is required.

8.6.2 Equipment

Equipment is needed to support the coordinating CSIRT staff, similar to the requirements for the internal centralized CSIRT. This includes (but is not limited to) the following:

8.6.3 Infrastructure

The infrastructure provides a secure environment for CSIRT day-to-day operations. This includes (but is not limited to) the following:

If the coordinating CSIRT is hosted by another organization, it can take advantage of some of its network infrastructure. Great care must be taken to ensure the confidentiality of incident and vulnerability data, therefore a firewall to isolate the CSIRT local network is highly recommended.

One of the most important infrastructure components needed for a coordinating CSIRT to interact with its constituency are formal, secure methods for collecting and disseminating computer security information, incident reports, vulnerability reports, and other alerts or warnings.

8.7 Summary

This model is fundamentally different from the other models described in this handbook, although many of the components and services may be similar to those previously discussed. Since a coordinating CSIRT is established to serve the interests of a larger constituency that potentially comprises hundreds of independent entities rather than a single organization,6 the manner in which services are delivered can be very different from the way they are provided by internal CSIRTs.

8.7.1 Impact on Constituency

Since the coordinating CSIRT is not usually involved in the actual recovery of systems or in securing compromised internal systems for the constituency, it can concentrate on coordinating activities between multiple independent parties and provide a level of neutrality not otherwise achievable. It maximizes the utilization of a relatively low number of staff in one strategic location and provides the central coordinating capabilities to allow a broad understanding of the security threats and activity affecting the constituency. It can quickly synthesize information available from a wide variety of constituent sources and disseminate it to the organizations in the constituency.

This team responds to reports of abnormal activity and incident reports, participates in incident and vulnerability analyses, and plays a proactive role in promulgating computer security awareness throughout the constituency. It also acts as point of contact for other CSIRTs that want to report incidents involving sites in the constituency. Coordinating CSIRT members collaborate and participate in security-related working groups or workshops, promote security awareness and training, and lend their expertise in testing and analysis activities.

The main impact on the constituency is to understand what type of interaction they can expect with the coordinating CSIRT, how and when to report information, and how to receive and follow any guidelines or recommendations coming from the coordinating team.

To be successful in its coordination role, the CSIRT must be trusted by the constituency, provide value-added services to the constituency, and have established points of contact and communication mechanisms for interacting with the constituency. These should include special secure communications technologies, use of encryption or authentication technologies, and specialized mail distribution lists. Having a complete and verified list of points of contact within the constituency will help determine who should be notified when information is distributed and will reduce the time needed to disseminate the information appropriately.

8.7.2 Constraints

The main constraints in this model are the difficulty of building effective relationships with all entities in the constituency and gaining their trust so that incidents are reported and recommended mitigation and prevention strategies are followed. Operating across a large geographical area with multiple time zones adds to the difficulties a coordinating CSIRT may face. If coordination takes place in an even broader context, differences in language, culture, and laws can create difficulties in providing an appropriate level of assistance to all involved parties.

Other constraints include ensuring that the coordinating CSIRT works together effectively with the organizations in its constituency. This is especially true as coordinating CSIRTs almost always have no authority over their constituency and serve in an advisory capacity, making it difficult to enforce any recommendations or guidelines, even when there are widespread attacks.

Because the coordinating CSIRT may not have direct authority, members of the constituency can choose to ignore its advice and recommendations. They can also choose to handle incidents on their own without reporting activity to the coordinating CSIRT. This can limit the amount of information the coordinating CSIRT has to work with in determining the scope, nature, and impact of any activity or threat.

Another constraint can involve the parent or hosting organization for the coordinating CSIRT. If this host organization does not have a trusted reputation in the constituency, this can affect how the CSIRT is perceived and cause constituents to fail to report to the coordinating CSIRT. Very often a coordinating CSIRT survives on its reputation, along with the accuracy and value of its services.

Finally, a problem may result regarding the expectations that the constituent members have versus the actual services offered by a coordinating CSIRT. The constituent may want a deeper level of service provided than the CSIRT is able to provide. For example, the constituent may want someone to come to their site to help in the recovery and response efforts, and this may not be a provided service.

8.7.3 Strengths and Weaknesses of the Model

The main strength of this model is that it provides a stable core of CSIRT professionals, in one central place, who are tasked with coordination. The full-time members provide stability, expertise, and a permanent infrastructure.

The greatest weakness to this approach is that the team might lack the operational knowledge and the ability to address the operational units in its constituency. If this issue is not handled well, it can result in a team that is not accepted and therefore does not receive incident reports and has little impact on the constituency.7

The strengths and weaknesses of this coordinating model include the following:

 


1 If the coordinating role is assigned within a group of organizations that have contractual or legal relationships, such as an industry group or holding company, stronger means of authority might be applied. For example, if the coordinating CSIRT reports to the board of the holding company, its advice might be presented to the organizations within the holding company in a way that has a great deal of authority, not defined by the coordinating CSIRT, but by the board.

2 For example: U.S. military (DOD-CERT); U.S. Federal Government (FedCIRC), country-wide CSIRTs such as the Singapore Computer Emergency Response Team (SingCERT); or CSIRTs that have research/academic networks as their constituencies, such as CAIS - Brazilian Research Network CSIRT (CAIS/RNP).

3 Most often teams that fit this category are centralized. In some cases distributed teams or combinations of distributed and centralized teams can be found. For this document we describe the model most often observed.

4 If the coordinating CSIRT is co-located with one of the organizations of the constituency, great care must be taken to not risk this neutrality.

5 For more information see <http://www.cert.org/kb>.

6 The coordinating CSIRT might be hosted in one entity that is also part of the constituency, but from a service provider point of view, the hosting organization is no different from any other entity in the constituency. There are differences in some cases; for example, attacks (such as DDoS attacks) on the hosting organization will affect the coordinating CSIRT as well, and vice versa.

7 Depending on the environment and other circumstances, support by experts from the constituency might be made available or arranged. This can reduce this particular weakness.

 

 


[Abstract]   [Title Page]   [Preface]   [Acknowledgements]   [1 Introduction]  
[2 Establishing CSIRT Capabilities]  [3 Operational Issues]  
[4 Security Team--Using Existing IT Staff]  
[5 Internal Distributed CSIRT]  
[6 Internal Centralized CSIRT]  
[7 Combined Distributed and Centralized CSIRT]  
[8 Coordinating CSIRT]  
[9 Choosing the Right CSIRT Model for Your Organization]  
[10 Closing Remarks
  [Appendix Summary of Services Offered [Bibliography]   [PDF File]