Organizational Models for Computer Security Incident Response Teams (CSIRTs)

[Abstract]   [Title Page]   [Preface]   [Acknowledgements]   [1 Introduction]
[2 Establishing CSIRT Capabilities]   [3 Operational Issues]
[4 Security Team--Using Existing IT Staff]   [5 Internal Distributed CSIRT]
[6 Internal Centralized CSIRT]   [7 Combined Distributed and Centralized CSIRT]
[8 Coordinating CSIRT]   [9 Choosing the Right CSIRT Model for Your Organization]
[10 Closing Remarks]   [Appendix Summary of Services Offered] [Bibliography]   [PDF File]

5 Internal Distributed CSIRT

5.1 Overview

In an internal distributed CSIRT model, referred to as the "distributed CSIRT" through the rest of this document, the team is composed of staff from other divisions or sectors of the enterprise who report to a central CSIRT manager. The CSIRT is a formally recognized entity and has been given the responsibility for handling all incident response activities. The team is considered "internal" because it is a team within a particular organization or company, so it is internal to the enterprise. It is different from the security team model primarily because of
(a) the existence of more formalized incident handling policies, procedures, and processes, (b) an established method of communication with the whole enterprise concerning security threats and response strategies, and (c) a designated CSIRT manager and team members who are specifically assigned incident handling tasks.

The CSIRT manager reports to high-level management, such as a CIO, CSO, CRO, or the equivalent. While the CSIRT manager has a "centralized" office (in organizational terms), the team members are scattered across the organization's geographic and divisional locations. Members of the team are chosen based on their experience and expertise with various operating system platforms, technologies, applications, and security practices. Team members include systems and business experts, network engineers, and others who have the needed functional knowledge.

The distributed CSIRT has full authority to analyze activities and shared authority to respond to incidents as they occur. No enterprise-wide action is taken or recommended without the approval of the CSIRT manager and possibly upper management such as the CIO. The team also has the authority for enforcing recovery and mitigation strategies with the approval and consent of the management. Divisional and functional unit managers are notified of any action to be taken in their areas and are involved in the decision-making process to determine how to implement a response.

The team has the authority to release enterprise-wide advisories and other documents, including best practices, response and recovery steps, and security updates. The team can also be involved in synthesizing and analyzing all IDS or other network/system/application logs. In a very large organization the CSIRT may only handle this type of analysis when an incident is escalated, and the initial log analysis would instead be done at a local level.

5.2 Supported Constituencies

This type of model is found in large, distributed organizations such as multinational corporations, government organizations, and educational institutions.¹ In most cases, small organizations would not be best served by this model.

5.3 Organizational Structure

There are many different ways a distributed CSIRT can be structured. The structure will depend on the size of the parent organization, the number of geographical locations where business functions are located, the number of systems and platforms supported, the number of CSIRT services to be offered, and the expertise of the existing staff.

In all structures the main function of the CSIRT manager is to coordinate the work of the distributed CSIRT. The manager should be located close to other high-level managers or the CIO/CSO, wherever the CSIRT reports. It is possible that some other members of the team may be co-located with the CSIRT manager. Depending on the work that is required, this might include some support staff or in some instances one or two analysts to help in the synthesis and dissemination of information. The CSIRT manager acts as a liaison to other parts of the organization such as upper management, human resources, legal counsel, public relations, or other appropriate groups. The CSIRT manager also is the main contact point for the team for any external organizations that want to communicate with the CSIRT. A designated backup for the CSIRT manager should be assigned and trained so the CSIRT processes can function properly when the manager is not available.

Team members are selected from existing staff and are assigned to devote a percentage of their work time to reactive and proactive incident handling issues. The percentage of effort they devote is negotiated with their supervisors and the CSIRT manager. The team members may contribute only part of their time to CSIRT work or could be assigned 100% of their time to this work. If the team members only perform CSIRT work on a part-time basis, they will report to two managers: the CSIRT manager for CSIRT work and their divisional or department manager for their normal day-to-day work. Team members can be system and security administrators, database administrators, researchers, network engineers, and any others with needed functional expertise. Other extended team members may include representatives from legal counsel, human resources, public relations, risk management, and law enforcement or criminal investigation groups. Any core or extended team members should have designated backups who have been trained and mentored to perform the required tasks and functions.

For this model to work there has to be clear understanding on the part of both management and staff that the distributed team members must stop working on routine tasks when they are needed to perform incident handling functions. Usually this model works best in an organization whose distributed departments or sectors share strong common characteristics that enable them to share staff. A problem that can result in this model is that, depending on how the team is presented to the organization, it might not be viewed as being responsible for incident response across the whole enterprise.² Another problem that may result is that during a crisis, the distributed CSIRT staff are often put in a difficult position. Along with being responsible for their CSIRT work, they are usually experts in their own routine work. So in addition to working on CSIRT tasks during a crisis, they will be heavily used to combat the local impact of the activity. This may make it difficult for them to handle all the work necessary to resolve the crisis. Management has to prepare for this situation by ensuring other staff have been cross-trained in both the routine and CSIRT work, so that more resources can be applied when such a situation occurs.

CSIRT staff duties include helping with analysis, encouraging security awareness among those in their business divisions, and implementing agreed-upon response and mitigation strategies in their divisions. There may need to be some hierarchical delineation of the team in large organizations, which might involve supervisors for platforms, divisions, or geographic areas.

The distributed CSIRT serves two purposes: (1) it provides a broad base of expertise across all the systems in the enterprise and (2) it gives the CSIRT a foothold in each division to not only coordinate activity but promote following best practice security policies and response steps. In this way, members of the team are out in the field (i.e., local sites); they are the eyes and ears of the CSIRT. They are also the arms and legs of the CSIRT, as they will be the ones to perform the response or provide guidance to those who will be performing the response. The distributed CSIRT staff have first-hand, real work experience concerning the operations and issues facing the organization. This brings a practical view of what techniques and approaches will work to mitigate problems.

The purpose of the CSIRT manager's office is to synthesize the reports received from multiple locations to identify trends and patterns of activity, and to help identify the scope and impact of any suspicious behavior or intrusion. The CSIRT manager also coordinates the work of the distributed team members, while providing direction and guidance for the team's security policies and procedures. In some organizations the CSIRT manager may be the information security officer for the organization and as such call the distributed team together whenever an incident occurs, to perform a coordinated response.

The organization must decide how many employees from each division should be included on the team. The organization must also decide how to assign the various CSIRT services. Services can be assigned to particular individuals, groups, or departments based on their expertise, job function, geographical location, or business unit. For example, incident analysis and response for threats and attacks against UNIX systems may be handled by the part of the IT department responsible for securing and maintaining the UNIX systems. On the other hand, the responsibility for handling these UNIX incidents may be assigned to the business units in which the UNIX systems reside. So if the UNIX system in question was located in the marketing department, the CSIRT team members in that department would provide the response. Another option is to have team members handle incidents for their physical location or geographic area. In this case some individual or group is responsible for all the incident, vulnerability, and artifact handling at that geographic or departmental location.

An organizational structure that can also work is to assign specific CSIRT functions to particular groups. For example, one unit might be responsible for analyzing vulnerabilities in Windows systems. Another unit might maintain a test lab for incident and vulnerability analysis, and still another might be responsible for developing and distributing communications, such as advisories or best-practice recommendations. Whatever assignments are made, staff will need training on the supported platforms.

It is extremely important that all members of the CSIRT know who has what skills and expertise on the distributed team so that they can be contacted for assistance when needed. CSIRT members also need to know when these other distributed team members are available. A shared calendar or list of operating hours and relevant points of contact for those times may be helpful. There must also be a clear notification and contact procedure that is followed when asking for assistance from other members of the team.

Communications across the team are extremely important in this model to ensure the efficient and effective operation of the CSIRT. The team will need to stay in touch through secure communications such as email, secure teleconferencing or phone conferencing, or a secure extranet or intranet. Virtual meetings should be scheduled regularly to encourage the feeling of a team working together. There should also be some type of regular face-to-face meeting, so members of the distributed team can get to know one another and share experiences or raise issues not easily addressed via phone or email. Discussion topics can also include reviewing organizational processes and procedures, service level changes or additions, strategic planning, and technical training. Management should look for ways to incorporate team-building activities into the work schedule. Ideas may include having members of the distributed staff attend conferences together or work on mock incident scenarios or other projects together. When staff members get together at face-to-face meetings, some social gathering or activity could be built into the agenda to let the team members have an opportunity to get to know each other on a more personal level to develop camaraderie.

It is important that response to a security event occurs quickly. If the lines of communication are too deep or hierarchical in structure, team members may not be able to affect an appropriate response. Some level of authority must be given to team members to act in a responsible way within the general procedures and guidelines of the CSIRT. They could be empowered to adapt or modify procedures or guidelines in certain situations (with an after-the-fact review by management to ensure that appropriate actions were taken), discuss any lessons learned, or determine whether any policies and procedures need to be updated or added.

For the distributed CSIRT model to be successful, the following elements are required:

There must be management buy-in and cooperation at all levels, especially from the business units where the distributed staff are located.

The CSIRT manager must know what expertise lies in each area and determine how best to assign tasks and actions to maximize overall team success. Clear lines of responsibility and communication must be established, along with identified backups for various CSIRT assignments and functions.

The team must understand a variety of technologies.

There must be a vigorous effort to engage the distributed CSIRT members in team building activities.

There must be clear policies and procedures for how incidents and vulnerabilities are reported and handled, including escalation, notification, and resolution procedures.

There must be a shared, secure infrastructure that the team can use for communication, incident tracking, and sharing information. Information and incident activity must be pulled from all areas of the organization and synthesized to provide an enterprise-wide view of security problems, incident trends, and response strategies.

5.4 Triage

In a distributed CSIRT, the triage process is important for providing an understanding of the scope of the reported incident activity. Organizations adopting the distributed model must also decide where to locate the reporting function for the CSIRT both physically (geographical and building location) and organizationally (department or division) within the enterprise. Various options exist, as described in Section 3.3.1, Triage.

The main decision point in this model is whether all reports come in to the CSIRT manager's office, either through a CSIRT help desk function within that office or a centralized organizational help desk, or if reports will first come into the local level to the distributed team member.

If reports come in centrally, the CSIRT manager will determine where the incident should be sent for analysis and response. A predetermined contact list of distributed team members and their locations, skills, and technical responsibilities is used to make the decision concerning where to send the reported incident.

If reports come into the local level and the triage process is done by the distributed team member, they must ensure that the report and any response is added into any incident tracking system and that the CSIRT manager is notified of the activity in case further actions are required throughout the enterprise.

The key in the distributed model will be to ensure that all incident activity is collected and tracked by the CSIRT manager's office, so that the impact and threat across the enterprise can be determined and also so that any trends and patterns can be identified. An incident tracking system accessible to all members of the distributed team will be required so they can update the status of or review any incident being handled at the local level. This shared system will provide the CSIRT members with access to incident information across the organization that may provide insight, warnings, or remediation strategies that may be useful at the local level. The incident tracking system should have the capability to allow different team members to record the distinct actions they have taken to analyze and resolve problems, particularly if different people will be working on the same incident.

5.5 Available Services

The following sections describe the types of CSIRT services that might be provided in a distributed CSIRT model. It is recognized that every team is different, so these are general descriptions based on observations of and discussions with other teams. The method in which the services are delivered assumes a certain level of infrastructure, staff, and equipment. These are described later in this section.

5.5.1 Core Services

The following tend to be the basic services provided by a distributed CSIRT. They are somewhat different from the baseline core services discussed in Section 2.7.4.

Alerts and Warnings

In a distributed model, all alerts and warnings coming into the CSIRT or parent organization from other security experts, vendors, or CSIRTs are received by the CSIRT manager or his or her designee. From there the alerts and warnings are disseminated to all members of the distributed team. Team members pass on the alerts and warnings to other system and network administrators, business managers, or security teams at their sites. General alerts and warnings that affect all members of the constituency are sent to a predetermined mailing list by the CSIRT manager or their designee. For this service to work efficiently there must be an up-to-date list of people and units to notify. This list should be maintained by the CSIRT manager with input from all the relevant areas where distributed team members reside. Input should also be collected from any newly defined areas or departments or constituency groups in the enterprise. This list must be verified and updated on a regular basis.

If alerts and warnings for the CSIRT's constituency need to be developed, these are assigned by the CSIRT manager to the individuals of the distributed team with expertise in the technology and mitigation strategies that need to be discussed in the alert or warning. Even if the alert or warning is developed in another part of the distributed team, it should be reviewed and possibly sent out from the CSIRT manager or his or her designee. There may be a need to work with a technical writer to produce the final versions of the alerts and warnings. If a technical writer is not on staff as part of the CSIRT, it may be possible to use staff with the needed skills from the constituency or parent organization. Whatever method is used to obtain technical writing assistance, this arrangement should be established in advance, so that the technical writer can be called upon as needed.

Incident Analysis

The distributed team members focus their analysis on the affected systems in their area of responsibility. The CSIRT manager's office correlates the incident activity across the enterprise to determine the scope of the activity, the impending threat, and the response effort required. The CSIRT manager's office also analyzes any reports to determine any intruder trends or patterns. Based on its understanding of the overall picture, the CSIRT makes recommendations for strengthening security across all of the organization's systems.

Incident analysis is performed by the members of the team who have expertise in the functional area, operating system, network, or application software involved in the incident. For newly reported attack types, the distributed team can collaborate on the investigations, pooling resources and expertise across the enterprise to help identify, analyze, and develop recovery measures.

Incident Response Support

The main focus of the distributed CSIRT is on providing the incident response support and guidance necessary to analyze and respond to incident activity. In most cases the distributed team members work with system and network administrators at the local level to help them respond to incidents, rather than performing the repair and recovery work themselves, as they would if they offered on-site incident response services.³

However, it is true that in some organizations, the distributed team members themselves are the system and network administrators and may actually perform the incident response tasks. Because this is not always the case, however (since some organizations use security officers or information security officers as their distributed team members and these people are not usually network and system administrators), the on-site incident response service is included in the next section, "Additional Services."

Incident Response Coordination

Coordination is handled initially by the CSIRT manager of the distributed CSIRT. This includes keeping each part of the distributed team up to date with the latest information, distributing information about the impact and scope of ongoing incident activity, and providing guidance for response strategies during events.

The CSIRT manager or designee is the main point of contact to coordinate any information dissemination or collaboration with upper management, the organization's legal counsel, human resources, law enforcement, or other internal parties unless organizational policies dictate that someone else must be that point of contact.

If other external parties such as victim or source sites, other external CSIRTS, or other security experts need to be contacted, the CSIRT manager or designee would also be responsible for orchestrating those interactions as appropriate, unless organizational policies determine someone else as the external point of contact. Those performing the triage function would act as an initial point of contact for such communication as well and pass information on to the appropriate team member.

In this model, CSIRT procedures are in place to escalate events to higher management, coordinate with public relations, or pass security events to law enforcement or other investigative bodies for criminal investigation as needed. The distributed team members understand the guidelines and serve as the points of contact for routing information to others in their division or geographical area as appropriate. Enterprise-wide messages, alerts, and advisories are sent from the central CSIRT office, upper management such as the CIO, or even public relations.

Because there is a coordinated triage function, information from across the enterprise can be reviewed. This allows the team an opportunity to identify any security gaps and determine the scope and potential impact of the reported activity. By seeing all the activity, the CSIRT can more easily prioritize and balance the workload. They are also able to predict or head off potential problems. For example if a virus is spreading across the network in one geographic area, it could be identified and stopped by proactively taking steps before it affects another geographic area.

Although there is information sharing among the CSIRT's members, without a focused and energized CSIRT manager, strong management support from divisional supervisors, and some quality assurance testing, there is no good way to ensure that all members of the team are reacting appropriately to assigned tasks. With a large, distributed organization, there must be a way to check that response steps are handled in a consistent manner. There must also be a follow-up mechanism in place to ensure that all response steps were implemented at each site as directed. Supervising these follow-up functions is one of the duties of the CSIRT manager. There will need to be sufficient resources to allow such work to occur.

Vulnerability and Artifact Response Coordination

If any vulnerability or artifact response coordination is undertaken, it is handled in a manner similar to incident response coordination. Information on the analysis and mitigation strategies and response efforts regarding any vulnerabilities and artifacts is consolidated at the CSIRT manager's office for dissemination to the rest of the team. For the most part, the actual analysis of any artifacts or vulnerabilities is done by the members of the distributed team with technical expertise in the affected operating systems and software, but it may also be done by an outside source such as a vendor or other external CSIRT. Whoever does the work will then pass their analysis or remediation strategies to the CSIRT manager for dissemination.

Because of the distributed CSIRT structure, even if vulnerabilities and artifacts are found on systems in one part of the organization, the analysis and response can involve team members from other geographic or departmental areas who have expertise to handle the required tasks. This coordination and any assignment of tasks is orchestrated by the CSIRT manager.

Even if no vulnerability or artifact response effort is undertaken by the distributed CSIRT, the team will still need information about any vulnerabilities or artifacts found in their systems. They will most likely look to other public or private information resources to get this information. This may mean getting alerts, advisories, or mitigation strategies from other external CSIRTs, vendors, or security companies. The CSIRT manager is the initial point of contact for work with other entities, but other distributed team members may be involved when their expertise is needed.

Announcements

Announcements are developed by the CSIRT manager or by assigned team members based on the topic and the team member's relevant expertise. Most often they are disseminated from the CSIRT manager or passed on to upper management or public relations for broadcasting. Assistance from technical writers may be required to ensure quality and understandability.

5.5.2 Additional Services

In addition to its core services, a distributed CSIRT may choose to offer other services. The following services are those most likely to be provided.

Incident Response On Site⁴

Since the distributed team members are located at various sites throughout the organization, and since they may actually be the system and network administrators at these sites, it is possible to have them perform the actual response. This service can only be provided if the distributed team members have the requisite skill set and the available time.

The distributed team members still receive directions and guidelines from the CSIRT manager, but would also need to have some level of authority to take appropriate actions during emergencies or when the threat is immediate. It is very important that the distributed team members pass all information gathered and all steps taken during an incident on to the CSIRT manager. Conversely, when the CSIRT manager's office disseminates a set of response steps or strategies to be implemented, the distributed team members need to confirm that they have executed the response correctly.

Vulnerability and Artifact Analysis

If this service is provided, it will probably be done on an ad hoc basis, initiated by a real need when artifacts are found on compromised or infected systems, or when a vulnerability is found in software supported within the organization. In a distributed team model, CSIRT staff members usually do not have the time or expertise to do this type of work for general research purposes. Distributed team members can, however, engage in vulnerability and artifact analysis to determine what impact any new vulnerabilities or found artifacts have on their infrastructure.

Whatever analysis is done, there must be some way to record and track the vulnerabilities and artifacts analyzed and the response that was taken to handle them. The CSIRT can also choose to store the artifacts found in some type of archive. In this way, any new artifacts can be compared against those in the archive to determine if they are similar or new, what they signify, and how to handle them.

Staff performing this service are required to update the rest of the CSIRT with the information discovered through their analysis. This helps other parts of the team perform the appropriate response if their systems are also threatened.

If analysis is not done, information about vulnerabilities and artifacts is obtained from other entities such as other external CSIRTs and security experts, as described under "Vulnerability and Artifact Response Coordination" in Section 5.5.1.

Vulnerability and Artifact Response

Just as in incident response on site, vulnerability and artifact response can be performed by the distributed team members at each organizational location. Again the staff needs to have the required expertise and understand supported platforms for the local site or across the enterprise.

If staff perform this function, they determine the appropriate actions to detect and remove artifacts found on systems. They search for and patch vulnerabilities. The CSIRT manager or other team members with the necessary expertise provide guidance.

In addition, distributed team members may take protective measures to avoid similar future attacks and incidents. This usually involves implementing secure configurations, updating or creating virus signatures that can be added to virus scanning databases or intrusion detection systems, and keeping operating systems up to date with new versions and patches.

If this service is not handled by the CSIRT, it is most likely handled by the organization's IT department, security team, or through a contracted managed security service provider (MSSP). The CSIRT will require established channels of communication to interact and share information with any other group providing this service.

Technology Watch

If done, this service is probably only performed at a cursory level, depending on available time and resources. If members of the distributed team have other work duties, they may not have time to work on a technology watch function.

If this service is performed, there are several ways it can be provided. In the more centralized method, the focus for this function resides with the CSIRT manager's office. This may mean that an additional staff member is needed in this office. In a more distributed method, one area of the team can be assigned this function as a full-time responsibility. Another distributed method is to have some members of the distributed team assigned to stay current on information in their area of expertise, such as a particular operating system, architecture, or function (IDS, firewalls, etc.), and designate one or more team members to consolidate the information from the other team members.

No matter what the method, staff members assigned to perform this watch function do so by monitoring newsgroups, mailing lists, other advisories, alerts, etc. Information from other members of the team in the divisions and sections is forwarded to the CSIRT manager to be consolidated and disseminated throughout the team. Collecting this information from distributed CSIRT staff members gives a more comprehensive overview of information from people with expertise in various applications, operating systems, protocols, and tools. More importantly, it reduces the duplication of effort so that all team members aren't reading the same set of resources.

This consolidated information highlighting current attacks, threats, response steps, and workarounds is made available to team members via the secured intranet or extranet.

Security Audits or Assessments

With its technical expertise and experience handling new vulnerabilities, real incidents, and artifacts, the distributed members of the CSIRT could participate with an audit or assessment team in the provision of this service, or provide input into the development of compliance criteria and requirements.

However, it is important that the involvement of the CSIRT does not create a conflict of interest. Team members should not audit their own systems and networks or their own division's systems and networks. They must also be objective and diplomatic, so as to create a level of trust between the CSIRT and the system and network administrators who maintain the hardware, software, and perimeter defenses. The goal is to have the system and network administrators feel comfortable in accepting guidance and recommendations from the CSIRT or any other auditing or assessment group.

Configuration and Maintenance of Security Tools, Applications, and Infrastructures

In most organizations, responsibility for configuration and maintenance of security tools, applications, and infrastructures falls to the IT department or designated network or security administrators. Although security infrastructure elements such as firewalls and IDS are sometimes placed within the responsibility of the CSIRT, this should be avoided, where possible, to allow the team to focus on incident management rather than maintenance.

However, it should also be recognized that if the CSIRT is not responsible for these types of services, the team's expertise and experience with security tools may be useful in providing advice and guidance to the organizational staff who are assigned to these functions. Establishing a good working relationship with system, network, and security administrators and the IT department will help make any necessary response efforts that require changes to systems, firewalls, or network logging smoother and more efficient.

If the distributed team members do not perform CSIRT work 100% of the time and have other assigned duties, it may be the case that they perform these configuration and maintenance tasks as part of their normal job functions.

Development of Security Tools

Based on their involvement with the configuration and maintenance of security tools, applications, and infrastructure elements, members of a distributed team may experience situations in which a specific solution is not readily available. In such cases members of a distributed team might develop tailored tools to provide a workaround or temporary fix to help satisfy such specific requirements. This development work can occur only if they have the necessary expertise or skills, and will be an outgrowth of their practical experience with the systems. Coordinating such developments with the rest of the CSIRT is important so that other parts of the organization can benefit from the results.

Intrusion Detection Services

If the intrusion detection service is not provided by the IT department, it can be provided by the distributed CSIRT. In this case the IDS is set up in each relevant division, ideally under the management (or supervision) of a member of the distributed CSIRT. Information is gathered in a standardized fashion and passed to the CSIRT manager's office for review, consolidation, analysis, and appropriate response. This ensures that patterns of activity across the enterprise are analyzed and responded to in a comprehensive manner.

One part of the distributed CSIRT may also be given the assignment to review all IDS logs, synthesize the results, and disseminate any alerts on abnormal activity to the relevant area of the team for investigation, analysis, and response.

Security-Related Information Dissemination

The CSIRT can establish a centralized web site (and if appropriate FTP site) to provide organization-wide access to appropriate security-related information. They can also use these sites to disseminate information from other external sources that has been tailored to the needs of the constituency in regard to supported technologies and software. Information can also be distributed via newsletters and mailing lists. This may be a difficult service to provide depending on the available staffing resources. If done, it may be at a minimal level only--making available copies of patches and security alerts.

Unless a particular set of team members is assigned the task of maintaining any of these broadcast mediums, the CSIRT manager's office will most likely be responsible for synthesizing any information for release. Information can be collected or written by CSIRT staff during any free time they may have or as a particular assignment. This provides team members a chance to be involved in other activities and provides a change from their routine work assignments.

Information disseminated includes current activity reports, threat trends and patterns, security awareness tutorials, incident reporting forms and guidelines, current updates on CSIRT developments, and any special security-related information on various applications, protocols, and security or attack tools.

Maintenance of this CSIRT site can be the responsibility of one set of team members or the CSIRT manager's office. If the CSIRT manager's office takes the responsibility, additional staff may be needed to handle the update and maintenance functions.

If the CSIRT site is not maintained by the team but by other parts of the IT group, then it will be important for the CSIRT to work closely with the administrators to ensure the server is adequately protected and that information is updated in a timely manner.

5.5.3 Impact on Security Quality Management

The amount of time that can be devoted to security quality management services will depend on the resources available from the distributed team members. In most cases, the CSIRT manager may provide input into these initiatives. Distributed team members with functional expertise, can be pulled into initiatives as time permits.

Distributed team members are responsible for promoting security awareness at their sites. They can do this by holding briefings, tutorials, or brown bag lunches to make relevant information or documentation developed by the CSIRT available to the organizational divisions.

The CSIRT will likely be asked for input in regard to implementations and maintenance of security solutions. This, as well as the expertise of CSIRT team members, can lead to the team's involvement in testing potential products. This can be done in various ways, from informal tests to formal evaluations. The testing can occur across the enterprise or can be done in response to a request by some department or unit. Based on their skills and knowledge, the CSIRT could also be involved in the development of business continuity and disaster recovery plans or could assist in the provision of security audits and assessments.

5.6 Resources

The following staff, equipment, and infrastructure resources should be considered when implementing a distributed CSIRT model.

5.6.1 Staff

The distributed CSIRT comprises a small, centralized management staff and team members who are spread across the organization:

CSIRT manager (reports to the CIO or other high-level manager)

one or more system administrators for the CSIRT infrastructure (dedicated or part of existing IT service)

one or more administrative support person(s)

one or more analysts (depending on the services offered, these analysts may help synthesize incident statistics, IDS data, security alerts, and technical documents)

distributed team members (number determined by parent organization based on the size of the constituency being served and the services provided)

The distributed CSIRT calls upon other adjunct staff that may be assigned to the CSIRT on an as needed basis, such as

technical writers

trainers and instructors

public relations staff

legal counsel and criminal investigators

other technical experts (administrators, managers, Windows/UNIX/mainframe experts)

For this model to work effectively, pre-arranged agreements will need to be established for how and when this additional staff can be called upon to provide assistance.

The services provided by the CSIRT are determined by the skills of the existing system, network, and security administrators in the organization and the requirements of the enterprise. It is expected that such skills would include, for example, hardware and software expertise in whatever technologies and functional business systems are supported throughout the constituency, including any systems, software, and applications developed in-house.

All members of the team need training on the operation and purpose of the CSIRT, along with technical training in normal incident handling activities. Backup staff should be identified in each unit where a distributed team member works so there is one backup for CSIRT work and one for the team member's regular, non-CSIRT duties.

A method for holding CSIRT meetings for all distributed staff is necessary to encourage a true team attitude. This may be done via a secure teleconferencing system or even an extranet. Periodically, face-to-face team meetings (where feasible) should be held to help the team get to know one another. This can be done at training classes or special off-site sessions. Of course, there should be a system of backup personnel to perform CSIRT functions while the team members are meeting.

The parent organization should promote distributed CSIRT positions as highly desirable and emphasize that such team members play an important role in the overall computer security infrastructure. These positions should be recognized and compensated appropriately. CSIRT distributed staff will gain a wider range of skills and experience from their involvement on the team. This can be a useful selling point for getting their home department or unit to give approval and support for their participation on the CSIRT.

If compensation for the added responsibilities associated with serving on the team are in the form of supplemental payment, this will of course mean that additional salary costs will be incurred. This does not include costs for charge-back of other adjunct staff. Overhead and other fringe benefits need to be considered as well.

The organization can also choose to outsource some of the response capability to a third-party contractor. This service can be provided as a recurring cost or on a fee-based schedule, depending on a number of factors, including the type of organization, sponsor, or service requested. For a distributed team, the third-party contractor would either provide human resources to augment the distributed CSIRT or cover services such as advisories, alerts, or IDS monitoring to reduce the team's workload. The main responsibility for decision making should rest with the organization, however, not the contractor.

5.6.2 Equipment

The distributed team members use existing computer equipment, peripherals, telephones, pagers, and other equipment. Staff can negotiate for the use of other equipment for testing if it is not available in their area. Additional equipment can be appropriated through the CSIRT manager and/or other financially responsible individuals. Access to a secure intranet or other communications mechanisms will also be required.

It is possible that if there is any additional staff located with the CSIRT manager, some additional equipment will be required. This could include (but is not limited to)

office space and furniture (supplies, copier, etc.)

computer equipment for day-to-day operations and activities

telecommunications systems, including stationary and cell phones, and pagers if
appropriate

home equipment and remote access, if appropriate

5.6.3 Infrastructure

The distributed CSIRT infrastructure should include access for all distributed team members to

a secured incident report tracking system

a secured repository for archiving all incident and vulnerability artifacts, such as exploit scripts and toolkits

secure communications channels, such as secured email, phones, video conferencing, faxes, intranet, or extranet as needed

It should also provide

physical security

protected power sources

network and host security

ideally, a separate CSIRT network for shared systems with a secure configuration and firewall, as well as VPN access for distributed team members

secure backups and storage of CSIRT data

mechanism for updating software and patches

virus protection and scanning

web tools

encryption technologies

Depending on the systems used, this infrastructure might require special client/server hardware and software at team locations and the CSIRT manager's office.

5.7 Summary

This model staffs the CSIRT by assigning responsibilities to designated individuals across the enterprise. These individuals become members of a distributed "virtual" team. The distributed CSIRT has a manager and may also have a small support staff located with the manager.

5.7.1 Impact on Constituency

The distributed team in essence becomes a conduit for collecting information across the enterprise and using this information to formulate strategic plans for securing the infrastructure and responding to any incident activity. The distributed team also provides a channel for disseminating alerts and advisories outlining preventative measures to take to protect the infrastructure along with disseminating any response measure for intruder activity. The distributed team members can also act to ensure that the appropriate steps have been followed. They can provide this information as feedback to the centralized team.

This model has several major impacts on the constituency:

Each organizational entity must be responsible for continually providing the appropriate resources (people with the right skills and experience, authority, access, etc.).

An infrastructure (described above) must be provided to support the team.

Effective management of this team will require coordination across many parts of the organization and vigorous efforts to involve all members of the distributed CSIRT in team-building exercises.

Technical incident training and mock exercises should be conducted and supported to maintain proficiency.

Methods must be established to allow information sharing among the team members so they can learn from and act on one another's data and experiences.

There are also possible impacts on the security of the organization or constituency. With a coordinated, distributed method for collecting and analyzing data and performing response, a better picture of the preparedness of the overall organization should be seen. Also, response time should be quicker and response efforts more consistent, ultimately leading to lower response and recovery costs, less damage from security incidents and a more secure environment.

This distributed CSIRT model represents a modest approach to the infrastructure investment required to begin collecting and analyzing security threat patterns throughout the enterprise. It provides a virtual network for the identification of threats and vulnerabilities, the dissemination of security information and the implementation of a coordinated response plan for managing incidents and threats. It is an improvement over the security team model.

5.7.2 Constraints

The main constraint for this model involves enabling the team to function as a whole when members are separated across geographic and organizational locations and administrative/management domains. Such separation can create many logistical problems. These problems include

having staff with differing levels of expertise across the organization who may not implement a response action in a consistent manner across the enterprise

not being able to coordinate information across multiple sites and people

possible conflicts in the prioritization and support of distributed team members' CSIRT work by management

possibly not scaling well in very large, geographically dispersed organizations

Because of these constraints, it is essential to have an effective CSIRT manager who works well with other division and organizational managers and is able to coordinate and supervise team assignments. The manager must be able to negotiate for additional resources when needed. The CSIRT manager's ability as a negotiator and ambassador are paramount to the success of this model. Organizations constantly reorganize; managers and units come and go. A major overhead for the CSIRT manager will be the constant update of who in the business units they need to work with to ensure they have access to appropriate staff. The larger the number of dispersed team members, the more difficult it will be for the manager to negotiate with all the involved organizational units.

If the distributed parts of the organization are located in separate affiliated companies or in other countries, there may also be difficulties in coordinating actions because of differences in languages, laws, policies, procedures, and time zones.

5.7.3 Strengths and Weaknesses of the Model

In this distributed model the responsibility for incident handling is assigned to appropriate individuals across the organization. With the proper software, training, and equipment, this model can provide incident reporting and incident analysis, and serves as a vehicle for formulating and deploying effective responses across the organization. By having such a coordinated process the organization is able to set policy, enforce standards, and implement incident handling activities enterprise-wide. Because the distributed team is composed of operations personnel at various locations, these individuals are very attuned to local operations and conditions. This close association with local operations can provide valuable input to the development of practical policies and procedures.

This model does have its weaknesses, especially if the team is composed of staff that have split responsibilities. Finding individuals with the appropriate experience, skills, and training who are willing and able to take on additional responsibilities may be problematic. Once found and trained, these individuals must be allowed to invest the time and energy to keep their skills and abilities current. If this does not happen then the appropriate commitment from the operating units may not be sustainable over time. Consequently the required skills and capabilities may not be available when they are needed most. This makes the incident handling capability only as good as each part of the distributed team. Also, effective management and coordination of the distributed team may become a problem, without a strong leader and appropriate upper management support.

The strengths and weaknesses of this model can be summarized as follows:

Strengths

There is a focused responsibility for performing CSIRT services.

The capability exists to coordinate incident reporting, analysis, and response across the organization.

There is a centralized tracking system and a centralized repository of incident data.

A consolidated and comprehensive view of the vulnerabilities and incident activity across the enterprise can be developed and any trends or patterns identified.

Staff is available at the divisions or functional business units to enact response steps or provide information and expertise in their relevant fields.

Since the CSIRT staff members are located within the divisions and functional business units, they understand the systems and software at those locations. They will have first-hand knowledge in some cases of the organization's IT infrastructure and will know what components are critical to the infrastructure. The CSIRT staff bring a unique, practical approach to the real-life operation of the organization and will understand what strategies are going to be effective and viable.

Weaknesses

Experienced staff may have commitments that prevent their participation in distributed team activities.

Existing staff may not have the required security training or expertise. Finding experts in the organization can be difficult, and over time there can be problems with turnover and training issues.

Staff members may be unwilling to take on the additional responsibility unless they perceive some value in the work that is being performed or receive additional compensation.

Determining where the CSIRT authority lies and the willingness of other divisions and functional units to accept that authority can be difficult.

Keeping the staff up to date with new technologies is difficult.

Keeping communications current and timely across a large, geographically dispersed area is difficult.

Verifying that CSIRT response efforts are implemented consistently across the enterprise can be difficult and time consuming.

Keeping the list of staff and their areas of expertise up to date is a difficult task for the CSIRT manager without the cooperation of the distributed team members and other groups within the organization.

If the organization is very large and crosses multiple time zones and countries, this model may not scale.

¹ This model especially can be found in commercial organizations with multiple sites and locations.

² To overcome this problem it is necessary to create an organizational image of the team as a single, tangible object, regardless of its virtual and distributed nature.

³ It is possible that the distributed team may indeed be structured to provide on-site response, as discussed under Additional Services.

⁴ This service is not included under core services because many teams only perform a support or coordination role and do not do on-site recovery and repair of systems. However, since some teams do provide this service, we've included it here.