The Critical Success Factor Method: Establishing a Foundation for Enterprise Security Management

Author
Richard A. Caralli

Principal Contributors
James F. Stevens
Bradford J. Willke
William R. Wilson

Technical Report
July 2004

CMU/SEI-2004-TR-010

Networked Systems Survivability Program
Survivable Enterprise Management Team

Unlimited distribution subject to the copyright.

[Abstract] [Title Page] [To the Reader] [Acknowledgements] [1 Introduction] [2 Background] [3 History of the CSF Method] [4 A CSF Primer] [5 Applying CFSs] [Appendix A: CSF Method Description] [Appendix B: Case Study 1: Federal Government Agency] [Appendix C: Case Study 2: Large Country Government] [Appendix D: Glossary] [References] [PDF File]

To the Reader

This technical report is based on the work of John Rockhart and his colleagues at the Center for Information Systems Research (CISR) at the Massachusetts Institute of Technology in the area of critical success factors and information systems planning.¹ In our research at the Software Engineering Institute (SEI) in the areas of enterprise security management and enterprise resiliency, we found broad applicability of Rockhart's concepts as an important tool in developing and deploying an effective approach to security management. The use of Rockhart's concepts for this purpose forms the basis of this technical report.

In this report, we introduce readers to the critical success factors (CSFs) concept and a corresponding method for developing a working set of CSFs that we developed at the SEI. More importantly, we discuss our use of CSFs as a means for framing and focusing the security strategy, goals, and activities of an organization. For background, the history and early uses of the critical success factor method in the field of information systems planning are presented. With regard to enterprise security management and enterprise resiliency, we discuss our recent application of the CSF method in fieldwork with customers using the Operationally Critical Threat, Asset, and Vulnerability Evaluation (OCTAVE) risk assessment methodology. The high-level steps we defined and applied to develop CSFs for these customers are codified in this report for further application and research. Finally, we discuss other ways in which the CSF method can be a powerful guiding and directing activity for the definition and improvement of enterprise security management processes and practices in organizations.

Depending on your level of familiarity with the concept of critical success factors, there are several ways to make effective use of the material presented in this report. To facilitate your use of this material, we suggest the following:

If you have no familiarity with the concept of critical success factors or the work of John Rockhart, you should read each of the sections of this report in numerical sequence.

If you are already familiar with the concept of critical success factors and are interested in our application of CSFs in the areas of enterprise security management and enterprise resiliency, you should begin reading this report at Chapter 5, "Applying CSFs," and continue with Appendix B and Appendix C, which describe our field experience using CSFs in customer engagements.

Finally, if you have familiarity with CSFs and are interested in obtaining a systematic method for developing a set of CSFs, refer directly to Appendix A, "CSF Method Description."

However you decide to read this technical report, it is our hope that you will see the potential benefits of deriving and applying critical success factors in your organization and will realize improvement in developing and deploying your organizational security strategy through this simple, yet powerful concept.

¹ Rockhart's concepts are documented in "A Primer on Critical Success Factors," published by the Center for Information Systems Research in June 1981 [Rockhart 81]. Our use of this material as the basis of our research has been granted by permission of the author.

Acknowledgements

The authors would like to thank members of the Survivable Enterprise Management team of the Networked Systems Survivability Program who helped in the production of this report by applying the CSF method in fieldwork with customers and graciously sharing their experiences with us.

The authors would also like to thank Julia Allen of the Practices, Development, and Training team for her review of this material and her considerable feedback. We appreciate her support and willingness to explore these emerging ideas with us.

We are also grateful to David Biber for his extensive work in creating the graphics that so appropriately illustrate our thoughts and concepts and to Pamela Curtis for her careful editing of this report.

We would also like to thank our sponsors for their support of this work. It has already had great impact on our customers' ability to improve their security programs and in our ability to transition new technologies in the area of enterprise security management and enterprise resiliency.

Last, but certainly not least, we would like to thank John Rockhart, whose work in the area of critical success factors is still viable today. His work improved information systems planning for many organizations, and we hope that our application of CSFs will have the same impact in the field of information security and enterprise security management.

[Abstract]  [Title Page] [To the Reader]   [Acknowledgements]   [1 Introduction]   [2 Background]   [3 History of the CSF Method]   [4 A CSF Primer]   [5 Applying CFSs]
[Appendix A: CSF Method Description]
[Appendix B: Case Study 1: Federal Government Agency]
[Appendix C: Case Study 2: Large Country Government]   [Appendix D: Glossary]
[References]   [PDF File]

Introduction

An organization¹ primarily exists to serve its stakeholders--the customers, employees, business partners, shareholders, and communities that benefit from the organization's existence and growth. The organization's mission embodies this focus by stating the organization's purpose, vision, and values. Stakeholders are best served when an organization operates in a manner that ensures the mission is accomplished.

Accomplishing the mission in a logical and systematic way requires the organization to develop a strategy. The strategy encompasses a set of goals or targets that the organization must achieve in a specific period of time. These goals are transformed into lower level tactical plans and activities to be carried out at various levels throughout the organization. This process of strategic planning provides a means for ensuring that the entire organization is focused on a shared purpose and vision.

Figure 1: Strategic Planning in Organizations

However, setting goals and developing plans to achieve them is only one factor in accomplishing the organization's mission. The organization must also perform well in a few key areas that are unique to its mission and to the industry in which it operates. In fact, failure to perform well in these areas may be a major barrier to achieving goals. These key areas can be described as a set of critical success factors--the limited number of areas in which satisfactory results will ensure competitive performance for the organization and enable it to achieve its mission [Rockhart 79].

1.1 Critical Success Factors

Critical success factors (CSFs) define key areas of performance that are essential for the organization to accomplish its mission. Managers implicitly know and consider these key areas when they set goals and as they direct operational activities and tasks that are important to achieving goals. However, when these key areas of performance are made explicit, they provide a common point of reference for the entire organization. Thus, any activity or initiative that the organization undertakes must ensure consistently high performance in these key areas; otherwise, the organization may not be able to achieve its goals and consequently may fail to accomplish its mission.

1.2 Enterprise Security Management

Managing security² across an enterprise is one of the many business problems that organizations must solve in order to accomplish their missions. Regardless of what organizational assets are to be secured--information or technical assets, physical plant, or personnel--the organization must have a security strategy that can be implemented, measured, and revised as the business climate and operational environment change. In the long run, the effectiveness of the security strategy depends on how well it is aligned with and supports the organization's business drivers: ³ mission, business strategy, and CSFs.

Figure 2: Alignment of Strategic Plan and Security Strategy

¹It is our intention to apply the term "organization" in this report universally to "for-profit" and "non-profit" organizations. While the bottom-line objectives may be different, we find no useful distinction between these types of organizations--both are in operation to accomplish a specific mission.

² Managing security broadly refers to the process of developing, implementing, and monitoring an organization's security strategy, goals, and activities.

³ Throughout this document we use the term "business drivers" to collectively represent the organization's mission, values, and purpose; its goals and objectives; and its critical success factors.

2 Background

The work of the Survivable Enterprise Management (SEM) team of the Networked Systems Survivability (NSS) program at the Carnegie Mellon Software Engineering Institute (SEI) is focused on helping organizations improve their capabilities for managing security across their enterprises. A primary objective of this work is to establish strategic planning and risk management as essential components of a security management program.

In this section, we document some of the lessons learned from our development and fieldwork efforts. In addition, we introduce the use of CSFs as an important element of an organization's strategic plan for security.

2.1 Lessons from OCTAVE

One of the primary functions of executive-level management¹ is to manage risk across the organization. An organization's security strategy and goals must be framed in the context of risk to get the attention of executive-level management.

Only those risks to critical assets that threaten the accomplishment of the mission are worth executive-level management's attention, and then only if the organization would be significantly impacted if the risks are realized.

A risk-based approach to security strategy and management enables organizations to direct their limited resources to the operational areas and critical assets that most need to be protected. Risks to operational areas and assets that can directly affect the organization's ability to accomplish its mission must be identified, analyzed, and mitigated. This perspective of "focusing on the critical few" is a foundation of the OCTAVE information security risk assessment methodology [Alberts 01].

In OCTAVE, this principle is put into practice by creating an assessment team that is composed of personnel from the organization who understand the organization's unique business drivers and conditions. Implicitly, these personnel are likely to consider the organization's mission when they decide which operational areas and assets to include in the risk assessment activity.

Identifying and focusing on the most important operational areas and assets is perhaps the most important activity that an organization performs when deploying a risk-based approach to security. However, as we have learned in our fieldwork with the OCTAVE method, this can be a difficult task in a large, complex organization particularly because there may be numerous operational areas from which to choose, each with its own set of important assets. An analysis team must apply their judgment in selecting the right areas and assets, and must ensure that their selection aligns with the business drivers of the organization. Failure to select (and validate) the right operational areas and assets can significantly diminish the value of a risk-based approach to security.

2.2 Challenges for Security Management

In the past three years, our research, fieldwork, and classroom interaction has provided much data regarding the challenges and barriers that organizations face in making the transition from vulnerability-based² to risk-based approaches to security management. Overall, we have observed that many organizations understand clearly that success depends on gaining the sponsorship of executive-level management and aligning security goals with the mission, goals, and objectives of the organization. In this way, security goals become an enabler of the organization's mission or strategy, rather than a burden or expense. However, our experience suggests that many organizations are ill-equipped to define their security goals, let alone to make an explicit connection between their security goals and the strategic drivers³ of the organization.

This is not unlike a similar challenge that has been faced by information technology (IT) departments in organizations. The acceptance of the position of chief information officer (CIO) as a legitimate executive-level partner to the chief executive officer (CEO) and chief financial officer (CFO) has been a more recent accomplishment in many organizations. Legitimizing this position causes the IT department to become a strategic partner of the organization, helping it achieve its mission more efficiently and effectively. Many well-known organizations have indeed proven their ability to be competitive, to grow, and to accomplish their missions through innovative and strategic uses of technology.

In the same way, an organization's security strategy must align with and enable its organizational strategy. But, with the increasing dependence of the organization's mission on information technology, security strategy must also ensure that the organization is resilient against attacks, particularly on technology, that could disable the mission.

Our conclusion is that a strong partnership is lacking between executive-level management and the parts of the organization responsible for setting and implementing security strategy. To assist our customers with this challenge, we began to search for ways that could aid in making this connection more explicit.

2.3 Addressing Challenges with CSFs

One of the ways in which IT departments have addressed these challenges (as early as the 1970s) is by involving the organization at large in their strategic planning process. This process--known by many names, such as business systems planning--explicitly takes into consideration the organization's key business processes and data to determine the technology needs of the organization. To further determine priority, these efforts also frequently include a direction-setting activity such as the development of CSFs. If the organization's accomplishment of the mission is tightly linked to its performance in a few key areas and the technology plan is based on enabling high performance in these same areas, the plan can enable the mission.

We drew upon the broad experience of the SEM team to address similar challenges for security management. At least one SEM team member had previously used CSFs in the development of an information technology plan. Other team members were also familiar with CSFs, and thus we began to explore the CSF method as a possible way to help our customers improve the focus of their security efforts. We began our investigation of the method specifically in response to the increasing number of questions and concerns of customers in their attempt to develop a scope for their risk assessment activities--selecting the right operational areas and critical assets to focus on. In our fieldwork, we also observed the value of the method for security management and strategy and goal development.

¹ In this report, term executive-level management is intended to refer to those personnel in C-level (e.g., CEO) positions, as well as their first-level senior managers (vice-presidents, executive directors, etc.). These personnel are involved in the organization's strategic planning process and are responsible for setting the direction and course for the organization.

² We describe a "vulnerability-based" approach to security as one in which the primary focus is to react to vulnerabilities (such as viruses or intrusions) as they are identified, rather than to take a proactive, strategy-driven approach to security. Vulnerability management is an important part of managing security but rarely is sufficient alone for securing a large organization or enterprise.

³ In this report, the term "strategic drivers" is used to refer to the important components of an organization's strategic plan: mission, objectives, goals, and critical success factors. These drivers may sometimes be referred to as "business drivers" or "organizational drivers."

3 History of the CSF Method

The concept of identifying and applying CSFs to business problems is not a revolutionary new field of work. It dates back to the original concept of "success factors" put forth in management literature by D. Ronald Daniel in the 1960s.¹ However, the CSF concepts and approach are still very powerful today and are applicable to many of the challenges being presented in the information technology and security fields.

3.1 Beginnings

In the late 1970s and early 1980s, organizations found themselves in the midst of an information revolution. The growth of information systems in organizations resulted in the production of significant amounts of information for analysis and decision making. The advent of the personal computer and the evolution of the field of information "systems" to information "technology" were indicators that the information explosion would continue.

John F. Rockhart, of MIT's Sloan School of Management, recognized the challenge that the onslaught of information presented to senior executives. In spite of the availability of more information, research showed that senior executives still lacked the information essential to make the kinds of decisions necessary to manage the enterprise [Dobbins 98]. As a result, Rockhart's team concentrated on developing an approach to help executives clearly identify and define their information needs.

Rockhart's team expanded on the work of Daniel to develop the CSF approach. Daniel suggested that, to be effective in avoiding information overload, an organization's information systems must focus on factors that determine organizational success [Rockhart 79]. For example, in the automotive industry, Rockhart suggested that styling, an efficient dealer organization, and tight control of manufacturing costs are important success factors [Rockhart 79]. Using success factors as a filter, management could then identify the information that was most important to making critical enterprise decisions. Accordingly, the underlying premise is that decisions made in this manner should be more effective because they are based on data that is specifically linked to the organization's success factors.

In 1981, Rockhart codified an approach that embodied the principles of success factors as a way to systematically identify the information needs of executives. This work, presented in "A Primer on Critical Success Factors," detailed the steps necessary to collect and analyze data for the creation of a set of organizational CSFs [Rockhart 81]. This document is widely considered to be the earliest description of the CSF approach. Our interpretation and application of Rockhart's approach, as documented in this report, is largely based on this description.

3.2 Evolution of the CSF Method

Most of the work in success factors performed by Rockhart and Daniel was focused on refining the information needs of executives. However, as a logical outgrowth of this work, Rockhart hinted at the usefulness of the method as a component of strategic planning for information systems or technology [Rockhart 81]. The CSF method has found its way into many formalized information or business systems and technology planning methodologies that are still being used today.

The CSF method and the analysis of CSFs have been used in many ways outside of the information technology planning arena. In their research on the use of CSFs in federal government program management, James Dobbins and Richard Donnelly [Dobbins 98] identify uses of CSFs to

identify the key concerns of senior management

assist in the development of strategic plans

identify key focus areas in each stage of a project life cycle and the major causes of project failure

evaluate the reliability of an information system

identify business threats and opportunities

measure the productivity of people

While this is not an exhaustive list of the ways in which Rockhart's original work has been applied, it suggests the broad applicability of the method. It speaks to the use of CSFs as a way for organizations to focus and validate many of the important activities they perform to accomplish their missions.

¹ Daniel's concepts are described in "Management Information Crisis," Harvard Business Review, September-October 1961.

[Abstract]  [Title Page] [To the Reader]   [Acknowledgements]   [1 Introduction]   [2 Background]
[3 History of the CSF Method]   [4 A CSF Primer]   [5 Applying CFSs]
[Appendix A: CSF Method Description]
[Appendix B: Case Study 1: Federal Government Agency]
[Appendix C: Case Study 2: Large Country Government]   [Appendix D: Glossary]
[References]   [PDF File]

4 A CSF Primer¹

CSFs are an explicit representation of the key performance areas of an organization. In this context, CSFs define those sustaining activities that an organization must perform well over time to accomplish its mission. They are found at every level of management, from executive to line management. Each organization also has a set of CSFs that it inherits from the particular industry in which it operates.

To apply the CSF method and to use CSFs as an analysis tool, it is important to understand how they relate to the organization's strategic drivers and competitive environment. This section provides a foundation for understanding CSFs and defines these important relationships.

4.1 CSFs Defined

The term "critical success factor" has been adapted for many different uses. Familiarity with the term is often presented in the context of a project or an initiative (i.e., the CSFs for the implementation of an ERP system or the deployment of a diversity program). In this context, CSFs describe the underlying or guiding principles of an effort that must be regarded to ensure that it is successful.

A slight distinction must be made when considering CSFs as a strategic driver at the organizational or enterprise level (as is done in this report). In this context, CSFs are more than just guiding principles; instead, they are considered to be an important component of a strategic plan that must be achieved in addition to the organization's goals and objectives. While this distinction is subtle, it is intended to point out that an organization's CSFs are not just to be "kept in mind"; their successful execution must drive the organization toward accomplishing its mission.

Many definitions of a CSF at the strategic planning level have already been provided in this report. In his seminal work on CSFs, Rockhart provides a useful summary of similar but distinct definitions [Rockhart 81]:

key areas of activity in which favorable results are absolutely necessary to reach goals

key areas where things must go right for the business to flourish

"factors" that are "critical" to the "success" of the organization

key areas of activities that should receive constant and careful attention from management

a relatively small number of truly important matters on which a manager should focus attention

The fact that CSFs can be defined in so many different ways speaks to their elusive nature. Managers generally recognize their CSFs (and the organization's) when they see or hear them, but may be unable to clearly and concisely articulate them or appreciate their importance. In fact, most managers are aware of the variables they must manage to be successful, yet only when problems arise and root causes are identified are these variables made explicit. For example, suppose an organization finds an alarming number of duplicate payments to vendors. They might conclude that this problem is related to poor staff training or high levels of staff turnover. As a result, the effective management of human resources (attracting, training, retaining) might be identified as an important factor that can impede the achievement of their strategic goals. In the process, they have explicitly defined a CSF for the organization.

CSFs are powerful because they make explicit those things that a manager intuitively, repeatedly, and even perhaps accidentally knows and does (or should do) to stay competitive. However, when made explicit, a CSF can tap the intuition of a good manager and make it available to guide and direct the organization toward accomplishing its mission.

4.2 Goals Versus CSFs

In traditional strategic planning and management, the definition of a goal or an objective is fairly well known; however, defining a CSF is much less clear [Rockhart 81]. Thus, CSFs are often confused with organizational goals. For the purpose of this report, we define organizational goals as targets that are established to achieve the organization's mission. They are very specific² as to what must be achieved, when it is to be achieved, and by whom. Effective goals have a quantitative element that is measurable to determine if the goal has been achieved. Goals can be decomposed into operational activities to be performed throughout the organization.

Figure 3: Goals vs. CSFs

Goals and CSFs go hand-in-hand. Both are needed to accomplish the organization's mission, and neither can be ignored without affecting the other. Because they are both integral parts of an organization's strategic plan, their relationship must be considered. For example, a person might have a goal of losing 10 pounds by the end of the year. To achieve this goal, the person would have to be mindful of a few key factors--improving his or her diet and nutrition, exercising regularly, and avoiding tempting social gatherings. Careful attention to these key factors will enable the person to achieve the goal of losing 10 pounds; conversely, inattention to these factors will inhibit achievement of the goal.

4.2.1 Relationship Between Goals and CSFs

The strong relationship between goals and CSFs results from the fact that managers are the origin of both goals and CSFs. When managers set goals, they also implicitly consider what they need to do to be successful at achieving the goals. Thus, it is likely that managers consciously consider their CSFs during goal setting and consequently create the bond between goals and CSFs that is needed to contribute to accomplishing the organization's mission. In this way, the influence of CSFs on goal achievement is made explicit, even if the actual CSFs are not. Organizations that have been successful at achieving their goals have also likely achieved their CSFs, albeit in a less observable way. Thus, goals sometimes resemble CSFs because they embody the importance of a key performance area.

Usually a goal is immediately discernible from a CSF because of its specificity. A CSF for the organization may be more general and is likely to be related to more than one goal. Consider the following goals for a large manufacturing company:

Increase sales in our Northeast division by 10% by 2^nd quarter, 2004.

Decrease travel expenses by 5% in the next 30 days.

Expand product line to include widgets and gadgets.

Increase expansion by opening at least two retail stores in at least two European markets by 3^rd quarter 2006.

The first goal might be commonly found in many commercial organizations: to achieve a 10% increase in sales in a divisional unit. To achieve this goal, the manufacturing company is stating an implicit dependence on the organization's ability to perform well in a few key areas. While the goal is simple, it reflects many key underlying assumptions or conditions. Implicitly, this goal states that

The growth of the company is dependent on the organization's capability for increasing sales.

Sales staff must be empowered and enabled to meet the challenge of attaining an increase of 10%.

The company must act quickly because it needs to retain and grow its market share in the Northeast as other competitors ramp up.

The Northeast division is an important area in which sales expansion brings the company a competitive advantage.

These assumptions or conditions embody CSFs that are directly related to the potential success in achieving the goal. For example, consider the following dependencies between the goal, underlying assumptions and conditions, and CSFs:

Figure 4: Relationship Between Goals and CSFs

The importance of the CSFs in helping the manufacturing company achieve its goals cannot be overstated. In this example, at least one of the CSFs--attract, train, and retain competent sales staff--is vitally important if the company wants to achieve the goal of attaining a 10% increase in sales. If the company fails to consistently retain qualified sales staff, the goal cannot be achieved, and in the long run, the manufacturing company's mission may be in jeopardy.

4.2.2 Cardinality³ Between Goals and CSFs

As illustrated above, an organizational goal may be related to more than one CSF to be achieved. Conversely, a CSF may influence or affect the achievement of several different goals. The potential many-to-many relationship between goals and CSFs is indicative of their interdependent nature and the importance of CSFs in helping the organization accomplish its mission.

4.2.3 The Superiority of CSFs Over Goals

Goals alone can be an unreliable predictor of an organization's ability to successfully accomplish its mission. This is because goal-setting in many organizations is at best a subjective exercise and often is strongly influenced by or derived from a performance management system rather than a strategic planning exercise. Often, goals are set with an eye to their achievability rather than how they contribute to accomplishing the mission. For example, an organization may realize that it is failing to accomplish its mission even though it has successfully achieved its goals. This can occur because the goals have not been aligned with the organization's strategic plan; thus their achievement does not propel the organization forward.

On the other hand, CSFs are less likely to be biased toward achievement. While CSFs are derived from and reflect the considerations of management, they are also inherited by the organization from the industry in which it operates, its position relative to peer organizations, and the effects of the current operating climate and environment. As a result, even though an organization may not achieve its goals, achieving CSFs may still get the organization closer to accomplishing the mission. Organizations that have achieved their goals but failed at their missions may have ignored the achievement of their CSFs.

The connection between an organization's operating environment and CSFs make them collectively more reliable as a predictor of the organization's capabilities for accomplishing the mission. To further develop this assertion, it is useful to explore the various sources of CSFs in more detail.

4.3 Sources of CSFs

CSFs are generally described within the sphere of influence of a particular manager. But there are many levels of management in a typical organization, each of which may have vastly different operating environments. For example, executive-level managers may be focused on the external environment in which their organizations live, compete, and thrive. In contrast, line-level managers may be concerned with the operational details of the organization and therefore are focused on what they need to do to achieve their internal, operational goals. Because of these different operational domains, the CSFs for the organization will come from many different sources. All are important for the organization as a whole to accomplish its mission, regardless of their source.

Rockhart defined five specific sources or types of CSFs⁴ for the organization as follows: [Rockhart 81]

the industry in which the organization competes or exists

an understanding of the organization's peers

the general business climate or organizational environment

problems, barriers, or challenges to the organization

layers of management

To provide an accurate picture of an organization's overall key performance areas, it is important to identify CSFs from each of these sources. However, as we found in our use of the CSF method, deriving CSFs at the highest levels of the organization tends to bring an acceptable mix of CSFs from many of these sources, so long as a broad cross section of management is represented in the process.

Each source of CSF and its importance to understanding the organization's key performance areas is discussed in more detail in the following sections.

4.3.1 Industry CSFs

Every organization inherits a particular set of operating conditions and challenges that are inherent to the industry (or segment of the industry) in which it chose to do business. This results in a unique set of CSFs that organizations in a particular industry must achieve to maintain or increase their competitive positions, achieve their goals, and accomplish their missions. For example, consider an organization in the airline industry. As a member of this industry, the organization inherits CSFs such as "deliver on-time service" or "move away from the hub-and-spoke system." Failure to achieve these CSFs may render the organization unable to stay competitive in its industry and may ultimately result in its exit.

Figure 5: Example of Industry CSFs for an Airline

Industry CSFs do not necessarily apply only to a commercial or profit-oriented mission. In reality, the concept of industry CSFs can apply to organizations that have a commercial, educational, public-service, or non-profit orientation. Thus the term "industry" in this context describes an organization whose purpose, vision, and mission is typically similar to those of its peers.

4.3.2 Competitive-Position or Peer CSFs

Peer-group CSFs are a further delineation of industry-based CSFs. They define those CSFs that are specific to the organization's unique position relative to their peer group in the industry in which they operate or compete. For example, an organization may be a leader or a laggard in a particular industry. If they are a leader, they may have CSFs that are aimed at ensuring they maintain or increase their market share against other organizations in the industry. On the other hand, if considered a laggard, the organization may have specific CSFs aimed at closing the gap and improving their competitive position relative to other organizations in their industry. In the case of the airline, an example of a peer-group CSF may be to "reduce cost per passenger mile" or "increase code share partnerships." These CSFs may be necessary for the company to increase market share in new geographical areas and to maintain or increase their competitive positions.

Figure 6: Example of Peer CSFs for an Airline

4.3.3 Environmental CSFs

To be successful, an organization must be mindful of the macro environment in which it operates. A closed organization--one that does not fully interact with its external environment--cannot survive in the long term. As a result, an organization must acknowledge the environmental factors that can affect its ability to accomplish its mission. Environmental CSFs reflect the environmental factors over which the organization has very little control or ability to actively manage. By making these factors explicit, the organization can at least be mindful of them and actively monitor their performance relative to them.

Environmental CSFs describe such conditions as current socio-political issues, the industry's regulatory environment, and factors such as seasonality. For example, the airline industry has been dramatically affected by terrorist activities, which have forced changes in airport operations and scheduling and have brought about new regulations with which airlines must comply. Unfortunately, airlines have very little control over this problem.

Figure 7: Example of Environmental CSFs for an Airline

4.3.4 Temporal CSFs

CSFs are tied to the long-term planning horizon of an organization. Over the strategic planning period the organization's CSFs may remain fairly constant, adjusted only when the organization makes major changes, such as changing its mission or the industry in which it competes. However, at one time or another, every organization encounters temporary conditions or situations that must be managed for a specific period of time, while continuing to maintain its performance in all other areas. These temporary conditions or situations can result in temporal CSFs--areas in which the organization must temporarily perform satisfactorily in order to ensure that its ability to accomplish its mission is not impeded. For example, the following conditions can create temporal CSFs:

threats that have been identified through SWOT⁵ analysis

temporary operating conditions, such as high inventory levels that must be reduced

extreme changes in the organization's industry, such as the effect of the 9-11 terrorist attacks on the airline and travel industries

barriers to entry to a new market or a new industry that arise when the organization takes on a new strategic direction

temporary environmental factors, such as war, extreme weather, loss of key employees

process or production problems that cause temporary changes in the organization's ability to produce its primary products or services

lawsuits or legal actions brought against the organization that must be managed as a course of business until resolved

Keep in mind that a temporal CSF may be an indication of a permanent change in the organization's industry, operating environment, or competitive position and as a result may be adopted as a long-term organizational CSF because of its strategic importance.

Figure 8: Example of Temporal CSFs for an Airline

4.3.5 Management-Position CSFs

Every layer of management has a different perspective and focus in the organization. This division of labor ensures that both tactical and strategic actions are taken to accomplish the organization's mission. Managers have different focuses and priorities depending on the layer of management in which they operate. This translates into a set of CSFs that reflect the type of responsibilities required by the manager's position in the organization. In fact, the CSFs that are inherent to the level of management may be universal across different organizations in the same industry. For example, executive-level managers may have CSFs that focus on risk management, whereas operational unit managers may have CSFs that address production control or cost control.

Figure 9: Example of Management-Position CSFs for an Airline Manager

Figure 9: Example of Management-Position CSFs for an Airline Manager

4.4 Dimensions of CSFs

In his initial work, Rockhart also described various dimensions of CSFs that are useful for understanding a particular manager's view of the world [Rockhart 81]. CSFs can be categorized by these dimensions to further clarify the current focus of the organization and how it is positioned among its peers.

The dimensions of CSFs as described by Rockhart are

internal

external

monitoring

adapting

4.4.1 Internal Versus External

Internal CSFs are those CSFs that are within the span of control for a particular manager. In contrast, external CSFs are those over which a manager has very little control. For example, in the airline industry example, an internal CSF could be "managing ground operations," while an external CSF may be "fuel costs."

Categorizing a CSF as either internal or external is important because it can provide better insight for managers in setting goals. For example, a manager can set very specific, achievable goals that complement the achievement of internal CSFs because the manager has control over them. However, if a manager has an external CSF, he or she must set goals that aim to achieve the CSF and minimize any impact on operations that may result because the CSF is not in his or her direct control.

4.4.2 Monitoring Versus Adapting

Monitoring CSFs emphasize the continued scrutiny of existing situations [Rockhart 81]. Because monitoring the organization's health is a primary function of management, almost all managers have some type of monitoring CSF. In fact, in our work with CSFs, we have found that many enterprise CSFs (those that apply to the entire organization) are focused on monitoring the organization's performance in a few key areas, such as compliance with regulations. Conversely, adapting CSFs are focused on improving and growing the organization. We have also found that many enterprise CSFs are adapting CSFs because they state the organization's desire to improve their competitive position or to make a major change in their mission. In these cases, the distinction between a goal and a CSF is less clear--what appears to be a goal of the organization is actually an adapting CSF.

4.4.3 Importance of CSF Sources and Dimensions

The source and dimension of a CSF provides additional information for understanding the importance of a CSF and its contribution to the accomplishment of the organization's mission. To be effective, managers must consider and monitor a wide range of activities, events, and conditions that occur throughout the organization and in the external environment in which the organization operates. Gathering CSFs that incorporate and reflect various CSF sources and dimensions provides an effective delineation of a manager's field of vision--a representation of the depth and breadth of the manager's responsibilities.

4.5 Hierarchy of CSFs

As explained previously, CSFs exist throughout all levels of the organization and can come from many sources. As with strategic planning and goal setting, CSFs at higher levels of the organization are related to (or dependent on) those at lower levels in the organization. Higher level CSFs cannot generally be achieved unless lower level CSFs are achieved as well.

Higher level CSFs influence lower level CSFs. In fact, if lower level CSFs differ significantly from higher level CSFs, the organization must consider whether there is proper alignment between the activities of lower level management and the strategic direction of the organization.

Goal setting also tends to follow a hierarchical pattern throughout an organization. However, in contrast to goal setting, there may not be a one-to-one relationship between CSFs as they cascade through the various layers of the organization. This is because CSFs are often closely tied to a particular manager or management layer and any specific concerns at that level. Thus, there may be some CSFs at lower levels in the organization that are important to achieving higher level CSFs and accomplishing the organization's mission but are not explicitly related or subordinate to a higher level CSF.

Figure 10: Example of Hierarchy of CSFs in an Organization

In our experience with CSFs, we have found it useful to describe two levels of CSFs: enterprise CSFs and operational unit CSFs.

4.5.1 Enterprise⁶ CSFs

The numerous sources of CSFs illustrate the broad array of challenges and demands facing management in modern organizations. Each layer of management has a set of conditions that must be monitored and acted upon. They also have a unique set of CSFs to consider.

But a simple gathering of the CSFs of each manager (and management layer) in the organization does not necessarily form a superset of enterprise CSFs. This approach could result in hundreds or possibly thousands of CSFs that the highest levels of management would need to consider. (Imagine the difficulties that strategic planners, for example, would have in attempting to align their planning activities with hundreds of CSFs.) It could also derail the organization's ability to focus on those five to seven areas that can truly "make or break" their efforts to accomplish the mission.

As with other managers in the organization, executive-level managers must be guided by their own set of unique CSFs. However, because of the role of executive-level management, their CSFs also typically represent the organization's truly critical and key areas of performance. This is not to say that the CSFs of other layers of management are not important--executive-level managers' strategic direction strongly influences the CSFs of other layers of management, and their ability to achieve enterprise CSFs is highly linked to success in achieving lower level CSFs.

Thus, an organization can develop a high-level set of CSFs that represent the top activities, concerns, strategies, and goals of executive-level management. These "enterprise CSFs" are derived from the top two or three layers of management and reflect the various CSFs found throughout the organization. In our work with CSFs, we have found that enterprise CSFs provide the most effective strategic view of what is important to the organization and to accomplishing the organization's mission. Enterprise CSFs represent the entire organization, and each operational unit in some way contributes to (or detracts from) achieving them by achieving its operational unit CSFs.

4.5.1.1 Nature of Enterprise CSFs

Enterprise CSFs often reflect both the current concerns of executive-level managers as well as the longer term strategic direction of the organization. As a result, enterprise CSFs can comprise a blend of temporal CSFs (reflecting the current "hot issues" of management) and industry, peer, and environmental CSFs (which reflect such indicators as the state of the economy, current business climate, and geopolitical issues). This is important because executive-level managers often must be agile and able to react to changes in addition to planning for the long run.

4.5.2 Operational Unit CSFs

An operational unit can be described as an organizational department, division, subdivision, or any other grouping of activities that share a common function, purpose, or mission. For example, the finance department in an organization might be an operational unit. Regardless of how organizations define their operational units, each may have its own set of CSFs.

As noted with enterprise CSFs, operational unit CSFs are not necessarily a simple collection of the CSFs of managers in the operational unit. Instead, operational unit CSFs may reflect the concerns and strategic direction of senior managers in the unit, as well as the strategic direction of the organization (as embodied in enterprise CSFs).

It is important not to confuse operational unit CSFs with management-function CSFs. Management-function CSFs reflect the generic responsibilities that are inherent in the manager's position in the organization. In contrast, operational unit CSFs are similar to enterprise CSFs in that they reflect the operating perspective and strategic direction of executive-level managers in the operational unit. The management layer is certainly a source of CSFs for the operational unit but is not entirely reflective of it.

4.5.2.1 Nature of Operational Unit CSFs

In our definition, operational unit CSFs tend to be less influenced by the organization's industry and more focused on the contributions necessary to support the organization's strategic goals and mission. For example, in the airline example, the operational unit CSFs for four divisions or departments--reservations, scheduling, flight operations, and freight operations--are very different, but each contributes vitally to the organization's overall achievement.

Operational unit CSFs may also have a temporal component, particularly if a specific division in the organization has temporary changes in operating conditions that it must consider. For example, if the airline industry as a whole must contend with overcapacity, the "scheduling" department may have a CSF that seeks to reduce flights and destinations served until demand increases.

4.5.3 Relationship Between Hierarchy and Source

Each of the sources of CSFs (industry, environment, etc.) can supply CSFs at the enterprise or operational unit level. However, because of their nature, some sources are more likely to supply CSFs at either the enterprise or operational unit levels. For example, industry CSFs may supply more CSFs to the enterprise level than to the operational unit level. Table 1 summarizes the possible relationships between enterprise or operational unit CSFs and the various CSF sources.

Table 1: Matrix of CSF Levels to CSF Types

CSF Level	Type of CSF
CSF Level	Industry	Peer	Environmental	Temporal	Management-Function
Enterprise	Industry CSFs strongly influence enterprise CSFs. Executive-level managers have a direct responsibility for interacting with the external operating environment of the organization as reflected in industry CSFs.	Executive-level managers must be mindful of the competitive position of the organization and calculate their role to ensure they plan accordingly.	Factors such as seasonality and the current geopolitical environment affect the current and long-term plans of the organization. Executive-level managers must consider the impact of the environment on their strategic plans.	A temporary problem or change in the organization's strategy can affect the overall CSFs for the organization. The hottest issues for executive-level management (such as security) must be considered and addressed.	Enterprise CSFs reflect the unique responsibilities of executive-level managers. Their position generally reflects their unique roles, such as risk management, financial management, and shareholder interaction.
Operational Unit	Industry CSFs could influence operational unit CSFs, especially if a particular division is affected. However, on the whole, there is less focus on the industry at this level than at the organizational level, particularly if the operational unit is fairly low in the organization.	Operational units may have less responsibility for the competitive positioning of the organization; therefore this may not be a source of CSFs. However, if the operational unit is a division that competes in a unique industry, competitive position CSFs will arise similar to those that could be found at the organizational level.	Environmental factors may filter down to an operational unit, particularly if it is a division competing in a unique industry, resulting in some environmental CSFs.	Temporary problems or changes affecting the organization as a whole may filter down to any operational unit that is critical to dealing with these problems or helping to implement changes. Therefore, some temporal CSFs may be found at the operational unit level.	Operational unit CSFs are highly influenced by management layer CSFs. Operational units tend to reflect many different unique layers of management (middle, line, etc.) and therefore are a rich source of management-function CSFs.

4.5.4 Other Considerations

Enterprise and operational unit CSFs must fit together and relate to one another, but they are generally much more loosely coupled than goals. Goals tend to cascade throughout the organization so that there is a tight one-to-one fit between the goals of each management layer. For example, the goals of a production line worker are directly related to the goals of the production line manager, whose goals in turn are focused on helping to achieve the goals of the chief operating officer and the organization.

The strict balancing and leveling inherent in goal setting is not typically found with CSFs. There may not be a one-to-one match between every operational unit CSF and an enterprise CSF. This is because each layer of the organization has its own focus and operating conditions, including executive-level management. However, there must be congruence; otherwise there may be a disconnection between what an operational unit views as important and what is good for the larger organization.

Figure 11: Relationship Between Enterprise and Operational Unit CSFs

¹ This section relies heavily on the description of CSFs as documented in the original primer by John Rockhart and Christine Bullen [Rockhart 81]. Their work is still widely recognized as the initial definition of CSFs and the CSF approach.

² Goals should be S.M.A.R.T.--specific, measurable, achievable, realistic, and tangible--to be effective. Goals that do not have this level of specificity can easily become confused with critical success factors. More information about the S.M.A.R.T approach to goal setting can be found in Attitude is Everything! by Paul J. Meyer [Meyer 04].

³ Cardinality refers to the extent of the relationship between two entities. A useful definition in the context of CSFs is "a business rule specifying how many times an entity can be related to another entity in a given relationship." (This definition can be found at http://www.vertaasis.com.)

⁴ In our application of the CSF method to security activities, we did not concern ourselves specifically with ensuring that CSFs were identified in each of Rockhart's categories. However, consideration of each of these categories makes a set of CSFs more robust and representative of all of the various operating domains of an organization.

⁵ SWOT analysis is a commonly used strategic planning technique. It identifies the organization's strengths, weaknesses, opportunities, and threats that should be considered in developing a strategic plan.

⁶ Rockhart refers to these types of CSFs generically as "corporate CSFs" because of the focus of his work on the corporate world. However, throughout this report, and particularly in the case studies, we use the term "enterprise CSFs" whenever we make a general reference to the critical success factors for an organization.

5 Applying CSFs

At the core, CSFs relate to the functions of management¹--what needs to be done, how well, and how often to meet a personal or organizational mission. In their simplest form, CSFs can be viewed as a management tool for making better-educated decisions that consciously support the mission of the organization. In fact, applying CSFs to validate and ensure alignment with the direction and intent of the organization can enhance any decision, initiative, effort, or process.

In this section, we describe the traditional uses of CSFs and some general advantages of a CSF-based approach to organization-wide efforts and initiatives. Most importantly, we explore the potential benefits of the CSF method as specifically related to addressing security strategy, goals, and activities. Finally, other potential uses of the method that we believe merit further research and field testing are presented.

5.1 Historical Application of CSFs

As noted in Section 3.1, much of the contemporary literature regarding CSFs (certainly that which postdates Rockhart's introduction of the CSF approach in the Harvard Business Review [Rockhart 79]) focuses on the connection between CSFs and information systems and technology. Even the creator of the concept, D. Ronald Daniel, had information systems in mind when he coined the phrase "success factors" and created the concept that Rockhart eventually transformed into CSFs. Ironically, Daniel's underlying objective was to help organizations manage more effectively; however, he quickly acknowledged that this was increasingly dependent on high-quality information and technology. Thus, the bond between CSFs and information systems was created and has continued to evolve.

5.2 General Advantages of a CSF-Based Approach

Throughout this report, the advantages of developing and applying CSFs are presented. The seemingly endless ways in which they can be of use to an organization speaks to their simple nature and broad applicability.

Of note is Rockhart's view that one of the most powerful uses of CSFs is to enhance communication among the organization's managers [Rockhart 79]. The ability to get managers "on the same page" can aid in mobilizing all areas of the organization toward the same goals. Regardless of how CSFs are used, there are several advantages to having this type of common focus for the organization:

CSFs can reduce organizational ambiguity. Developing and communicating a set of CSFs can reduce the dependence on the perceived aims of the organization. CSFs reflect the implicit, collective drivers of key managers and as a result are a more dependable and independent articulation of the organization's key performance areas.

CSFs are more dependable than goals as a guiding force for the organization. An organization can set good goals that, in theory, will move the organization toward its mission. However, if the goals are poorly articulated or developed, this is not guaranteed. CSFs are reflective of what good managers do well to move the organization toward its mission, regardless of the quality of the goals that have been set.

CSFs are more likely to reflect the current operating environment of the organization. Goal setting tends to be a cyclical (i.e., yearly) activity that is seldom revisited until performance measurement. Used properly, CSFs are likely to be more dynamic and to reflect current operating conditions (particularly because of the many sources of CSFs).

CSFs provide a key risk-management perspective for the organization to consider. The risk perspective of executive-level managers is built into CSFs, so their "radar screen" is exposed to the organization as a whole.

CSFs can be valuable for course correction. When CSFs are made explicit, managers often realize that their perception of what is important to the organization may not match reality or they may realize that they don't fully understand the current operational climate. Thus, they can use CSFs to realign their operating activities.

5.3 Using CSFs in a Security Context

Our interest in the CSF approach evolved from our recurring observation that customers often have difficulty developing and implementing a security strategy when they do not maintain an explicit focus on business drivers. This can occur for a number of reasons:

The organization may have decided that security is the domain of the information technology department, which may not play a strategic role or is unable to articulate the overall goals of the organization.

Security is viewed as a cost or burden that must be managed and not as an activity that contributes to success, profitability, or growth.

Personnel in charge of security are disconnected from the organization's mission because of their role or function (i.e., they are external to the organization, as with consultants, or they have a strict technology focus) or because of the layer of the organization where they operate (i.e., staff or line functions).

The organization's business drivers or factors for success simply are not well known or communicated to all who have a need to know.

Regardless of the reason, the result is often the same: the security strategy fails to reflect what's important to the organization, to the accomplishment of its mission, and to its long-term resiliency. It fails to answer the basic questions: What is to be protected? How is it threatened or why does it need to be protected? What happens if it is not protected? Certainly, these questions are fundamental to a risk management approach to security, but the answers are often embedded in the organization's mission, goals and objectives, and the factors that affect the organization's potential success or failure in pursuit of the mission and goals--the CSFs.

Unfortunately, many organizations with whom we have worked have only a vague understanding of their CSFs. They often rely on their perception of "important" or "critical" rather than relying on an explicit articulation of these factors. They also tend to rely on external influences (such as laws and regulations) to provide them with a default security strategy or initiative instead of developing an internal strategy, consistent with their mission, that can position them to address ever-increasing and changing regulations.

Overall, it is our contention that organizations that have a clear "eye on the prize" are better positioned to make meaningful decisions about security and to implement them in a way that not only protects the organization but actually contributes to the accomplishment of the mission. Properly positioned and managed, organizations can turn the burden of security into a competitive advantage--an enabler that directly affects an organization's achievement of its goals and its bottom line. Some organizations have had to adopt this perspective on security because it is required by the nature of the industry in which they compete. For example, the business model for many e-commerce organizations is built on trust and security. Thus, their security strategy is inextricably linked to their mission--if the strategy is effective, they meet their goals; if not, the bottom line suffers.

In this section, we provide some of our theories and share our experiences regarding the use of the CSF method to enable the effective development of security strategy and the application and management of security throughout an enterprise.

5.3.1 Enterprise Security Management

Several years ago, we were called upon to assist a federal government agency in its security efforts. The agency had recently decided to develop its own information security capability, through which it would not only serve itself but several other high-profile government agencies. Our scope of work was to perform a risk assessment for the agency to identify the issues that it would need to address first. However, it soon became clear that a risk assessment activity would not answer some of the basic questions and issues the agency needed to confront.

A team with a broad array of technology and security skills was assembled to staff the information security capability. However, what the agency had in terms of human resources did not compensate for what it lacked in other key ingredients for success--there was no existing security policy or strategy, no shared vision or objectives for strategy across the various agencies, and, more importantly, no clear vision of what it wanted to accomplish and why. In addition, the team appeared to lack clarity on its role and responsibilities.

Our work promptly took the form of helping the team to determine its security goals and objectives and to take an inventory of its strengths and challenges. The team members understood that they needed to "secure the organization" but were not able to clearly articulate the meaning of "secure" and, further, how they would know when they had accomplished it.

We observed that, as a newly formed group, one of their major challenges in defining "secure" or "security" was that the team lacked context--members had no comfort or familiarity with the mission of the larger agency or the missions of the other very diverse agencies that they were charged to protect. Before our work progressed any further, we suggested that it might be a good idea to collect these agencies' mission statements and study them to get a sense of what was important. This information could then help to determine the capabilities that the team would need to meet its requirements for managing security across such a vast enterprise.

In hindsight, what we were attempting to do was to get the agency to set the context for its security efforts--to develop a guiding "position" or a "posture" as we described it at the time. We prompted the agency to look clearly and explicitly at the drivers used by the organization to accomplish its operational goals and to align its security strategies and activities to those drivers. In that way, agency personnel might not only be supporting but contributing to the operational goals through their work. While we didn't perform a CSF exercise with the agency, it became clear to us that in the future, this type of exercise would be a valuable context-setting exercise for customers facing similar problems.

It also became apparent during our engagement that the small security staff that the agency had assembled would not be able to accomplish its security goals alone. It would need to draw upon and mobilize existing capabilities of the organization, both technical and managerial, to be successful.

5.3.1.1 Enterprise Security Management Defined

Our experience with this federal government agency (and subsequently several other organizations) evolved into a management- and process-oriented view of security as a business process that is pervasive across and dependent on the enterprise. Our continuing exploration of these theories is the focus of an emerging body of work in the Networked Systems Survivability program at the SEI, referred to as enterprise security management (ESM). The core assertion of this work is that managing security across an enterprise is a complex endeavor that depends on several fundamental principles:

The skills, capabilities, and efforts of the entire organization must be utilized and mobilized.

Key functions and processes in the organization must collaborate on shared security goals and strategy.

The organization's security objectives or an articulation of its "desired state" must be developed and understood.

Critical assets that are essential to achieving the organization's mission must be identified and protected.

Information technology operations and support must enable security goals.

One of the keys to achieving such an extensive undertaking, particularly where many diverse parts of the organization must work together, is to ensure that it is properly focused on a shared understanding of organizational values--such as CSFs.

5.3.1.2 ESM and CSFs

The complexity of undertaking an enterprise-wide view of security management can be illustrated in the challenges facing chief security officers (CSOs). Often, CSOs are tasked with "securing" the organization, but may not be clear on what that means. Indeed, in some organizations, the role of the CSO has been relegated to the information technology department, further separating it from organizational strategy and business drivers. As a result, the CSO is often left to answer some very important organization questions without specific guidance:

What needs to be secured? Why, and in what priority?

What parts of the organization must be involved in this effort? How will I convince these units to work together, especially if I don't have direct control over them?

How will I know when the organization has been "secured?" What will be used to measure success?

Our assertion is that some of the answers to these important questions are found in the organization's business drivers, and in particular its CSFs, because they represent a common, shared focus. Why?

The "field of vision" of top management (and management in general) is represented in CSFs. This provides a powerful clarification of what is important and valued in the organization. Failure to achieve CSFs directly affects the organization's ability to accomplish its mission. Thus, security efforts need to align with CSFs and ensure that the accomplishment of CSFs is not impeded.

CSFs reflect the goals of the organization. Managers operate toward the achievement of goals. What needs to be protected in the organization can be identified relative to these goals--assets and processes that support these goals and the organization's mission must be protected.

Rallying around a common purpose is an effective means for getting disparate parts of the organization to take on a common cause, such as security. Security is a business problem that requires the effort of everyone in the organization to solve and to manage. CSFs provide a unifying effect, if only because most employees prefer to avoid the stigma of failing to contribute to an effort that is clearly good for the organization.

The drivers for security should be the same as the business drivers used by the organization to accomplish its mission. Security should be a way for organizations to enhance their operations, help them achieve their goals, and provide them with an appropriate level of resiliency commensurate with their long-term strategies. CSFs can be shared drivers for security and the organization.

For these reasons, we see great promise for the CSF method as a catalyst for setting the direction of an organization's enterprise security management activities. Chief security officers can confront the challenges of enterprise security management by using CSFs as a foundation from which security professionals and the rest of the organization can collaborate, plan, and execute. They can also qualitatively measure the success of their security programs by determining how they contribute to achieving the organization's enterprise CSFs.

5.3.2 Information Security Risk Assessment and Management

One of the key activities in managing security is to perform periodic risk assessments. In general, risk assessments are a diagnostic tool that helps the organization to determine the success of its security efforts relative to its security strategy. The CSF method shows particular promise in helping organizations conduct more meaningful (and valid) information security risk assessments in a number of areas.

Most of our fieldwork experience in information security risk assessment is in the use and application of the OCTAVE² method. The OCTAVE method provides specific guidance for the major activities of a risk assessment, but also allows for significant tailoring to meet the needs of unique organizations. As a result, many users with whom we have worked have asked us for additional guidance on developing scope, selecting critical assets to assess, and in prioritizing risks to mitigate. Without the advantage of the CSF method, we often provided no specific guidance to customers except to encourage them to align risk assessment activities with business drivers. However, the term "business drivers" is often ambiguous and subject to interpretation. Unless an organization has a clear definition of its business drivers, they cannot be used in a practical way to guide important organizational efforts or initiatives.

Because of this issue, we began to search for a more precise and practical way to apply the concept of business drivers to security. Through further research and fieldwork, we decided to explore the use of CSFs. CSFs are inextricably linked to and representative of the other components of business drivers (i.e., the organization's mission, values, and purpose and its goals and objectives). CSFs are also a conduit to achieving the organization's goals and objectives and accomplishing its mission. Thus, the use of CSFs can be an effective way to link business drivers to various aspects of security, including developing and implementing security strategy, managing security activities and operations, and conducting security risk assessments. On this premise, the following sections highlight the ways in which CSFs can enhance key risk assessment activities.

5.3.2.1 Determining Risk Assessment Scope

One of the most important (and difficult) tasks in performing a risk assessment is to determine its scope. A risk assessment performed on an area of the organization that is not essential to accomplishing the mission generally will not yield meaningful results. Unfortunately, failing to properly scope the risk assessment also diminishes the purpose and intent of using a risk-based approach.

For example, the OCTAVE method for risk assessment guides users to choose three to five important operational areas to include in the scope. This guidance is perfectly acceptable for users who have a good sense of the organization's mission and can be objective about which areas contribute most to accomplishing the mission. However, for many users, particularly those in the lower levels of the organization, this guidance is difficult to put into practice. Frequently, users need an explicit set of criteria against which to evaluate operational areas and to decide which areas should be included in the risk assessment. CSFs are useful for this purpose because they represent the organization's business drivers and they embody the risk-management perspective of executive-level management.

Using CSFs, an affinity analysis³ can be performed between enterprise (or operational unit) CSFs and the various departments or operational areas of the organization being considered for assessment. Those operational areas that provide significant support for the achievement of CSFs will be strong candidates for risk assessment because of the implied contribution they make toward accomplishing the organization's mission.

Figure 12 provides an example of the possible intersections between enterprise departments and CSFs for the purpose of identifying areas in which to perform a risk assessment.

Figure 12: Affinity Analysis for Determining ISRM Scope

5.3.2.2 Selecting Critical Assets for Assessment

A risk-based approach to security encourages organizations to direct their limited resources to protecting the organization's most critical assets--information and technical⁴ assets that are essential to supporting the organization's mission. The selection of critical assets for risk assessment is often left to the judgment of those performing or participating in the assessment, whether they are inside or outside of the organization. Thus the importance of the asset may be based on its perceived value, rather than a more concrete method of asset valuation. While desirable, assigning a qualitative or quantitative value to assets may be prohibitively expensive for an organization.

The use of CSFs can be a simple yet effective compromise for selecting critical assets. As a byproduct of using CSFs to help define the scope of a risk assessment, the pool of potential assets can be effectively limited to those operational areas that are most important. Conversely, for organizations that have a solid inventory of information and technical assets, affinity analysis can be performed to compare assets to CSFs. The result of this type of analysis is the identification of assets that are essential to achieving CSFs and, by default, to accomplishing the mission of the organization. In summary, CSFs can help to validate the importance of an asset by confirming its overall significance to the organization.

Figure 13 portrays an example of affinity analysis between critical assets and a set of enterprise CSFs. In this case, there is an intersection between the "financial data" asset and the "manage compliance" CSF. This indicates that the "financial data" asset is critical to the organization because it is essential to achieving the "management compliance" CSF, and thus needs to be protected.

Figure 13: Affinity Analysis for Determining Critical Assets

5.3.2.3 Identifying and Validating Security Requirements

An important component of protecting critical assets is the development of security requirements in the areas of confidentiality, integrity, and availability.⁵ As an asset is stored, transported, and processed throughout the organization, these security requirements must be met and protected by all who use or take custodial control of assets. Defining security requirements can be a difficult task; significant thought must be given to the potential misuse of the assets and the consequences of this misuse. In addition, a substantial number of requirements could be developed for each asset. This poses a problem for devising a protection strategy for an asset: Which requirements are most important? Which requirements, if unmet for any reason, would impact the owner of the asset or the organization as a whole? Further, which assets, if impaired, would impact the achievement of CSFs?

Answering these questions requires consideration of the priority of the security requirements. CSFs can be very useful for this purpose because they represent management's priorities. For example, a comparison of an asset's security requirements to CSFs will highlight those requirements that are essential to ensuring that the achievement of CSFs is not impeded. Prioritizing requirements in this manner can help the organization to develop and implement meaningful security controls for assets to ensure that they continue to contribute to the organization's pursuit of its mission.

Figure 14 provides an example of affinity analysis for security requirements. In this example, the security requirement of "confidentiality" for the "medical records" asset has been identified as important to the "manage compliance" CSF. This is because failure to meet the confidentiality requirement for medical records could impede the organization's ability to be successful at managing compliance activities.

Figure 14: Affinity Analysis for Determining/Validating Security Requirements

5.3.2.4 Identifying Risks to Critical Assets

Risk identification is at the core of a risk-management approach to securing critical assets. Properly characterizing a risk is essential to understanding the potential impact on the owners of the asset if it is somehow compromised, temporarily lost, or permanently destroyed. While this task is essential, it can also be the most elusive for an organization to undertake. As noted previously, defining the scope of a risk assessment and determining the critical assets on which to focus the assessment is an important first step. However, the organization still has to decide upon which risks to direct limited resources. To do this, an organization has two options:

Use a generalized taxonomy to identify risk. This approach is popular with federal government agencies and is often effective because it provides an orderly and somewhat comprehensive guide for examining many potential areas of risk.

Elicit risk information directly from the organization. This is the approach used by the OCTAVE method and, depending on the organization, can also be very effective. It attempts to ensure that the experience and intuition of managers and staff in the organization is relied on to identify risks that are most associated with the business drivers of the organization.

While effective, there are potential problems with each of these approaches. For example, exclusively using a taxonomy may cause the organization to overlook certain risks that are unique to its business environment or to spend valuable time considering risks to which it is not specifically exposed. In addition, success in using a knowledge elicitation approach is highly dependent on ensuring that the right participants are interviewed and that they fully understand the risk assessment approach and objectives. While it may be effective in identifying risks that are unique to the organization, this approach can result in overlooking many common risks that the participants are not familiar with because they have a limited understanding of information, technical, and physical security issues. Thus, the results from this approach are only as good as the quality of the participants in the process.

One way to enhance the effectiveness of either of these approaches is to use CSFs. For example

CSFs can be used to properly focus risk identification. With a taxonomy approach, CSFs can help to focus in on those areas of the taxonomy that directly affect (encourage or impede) the accomplishment of CSFs. In this way, the taxonomy is more effectively linked to the organization's business drivers and areas that are unimportant to the organization are not considered.

In the case of the knowledge elicitation approach, CSFs can be a very powerful means for shaping and guiding the responses of participants. Knowledge of enterprise (or operational unit) CSFs can enable participants to identify areas of concern and risks that explicitly consider the potential impact on achieving CSFs. In this way, the participants are providing information that is more certainly linked to the organization's business drivers. (This is illustrated in the case study presented in Appendix B.)

Likewise, once risks have been identified, CSFs can be used for validation. Risks to critical assets that do not impair the achievement of the organization's CSFs may be given a lower priority because they are unlikely to impact the organization's ability to accomplish its goals and mission. As a result, risks that interfere with the organization's ability to achieve CSFs can then be focused on because they have the greatest potential for harm.

5.3.2.5 Setting Evaluation Criteria for Measuring Risk

In most commonly used risk assessment methods, a set of criteria is used to evaluate the extent of risks to critical assets. In risk assessment methods such as OCTAVE, the risk evaluation criteria is developed by the organization so that it uniquely reflects their business drivers and conditions; in other methods, the extent of risk is standardized in that the developer of the method has defined and weighted the criteria used for evaluating risk.

In our experience, a risk assessment is more meaningful when it is based on and connects directly to an organization's unique business drivers. Risk evaluation criteria that are developed by the organization are likely to reflect the values of the organization, but this is not guaranteed. The validity of the criteria is dependent on a number of factors, including: Who developed the criteria? What is that person's role (or perspective) in the organization? What is that person's level of familiarity with the organization's business drivers? This can be particularly problematic when risk assessments are performed at the operational unit level--evaluation criteria that are important to the unit may not be in synch with the organization's business drivers. Thus, the consequences of risk are only measured with respect to the unit and not the organization as a whole.

CSFs can be used to mitigate some of these issues with evaluation criteria. For example, affinity analysis can be performed between CSFs and the impact areas being considered for inclusion in the risk evaluation criteria. This comparison is a means for validating that the evaluation criteria accurately reflect what is important to the organization. As a result, there is more assurance that the evaluation criteria being used in the risk assessment will reflect a more accurate representation of risk.

Figure 15 shows affinity analysis for validating evaluation criteria. In this example, the organization has decided that the impact area "productivity" is directly related to its ability to meet the "continually improve operational efficiency" CSF. Consequently, any risk that impacts the organization's productivity also impacts its ability to successfully meet this CSF.

Figure 15: Affinity Analysis for Validating Evaluation Criteria

5.3.2.6 Evaluating Threats and Mitigating Risk

Organizations are vulnerable to many different threats and risks. Which threats should an organization be concerned about? Which risks need to be mitigated? The purpose of applying a risk-based approach to assessment is to focus on only those threats and risks that could have a significant impact on the organization. Implicitly, a risk impacts the organization by impeding its ability to conduct its normal course of business and to achieve its goals. For example, a risk that results in negative publicity impacts an organization by interfering with its ability to keep its customer base, attract new customers, obtain financing, etc. However, the organization is really impacted only if these consequences affect business drivers--goals, objectives, mission, and CSFs.

Comparing threats and risks to CSFs identifies those that are strong candidates for mitigation. Thus, as an important component of business drivers, CSFs can help an organization to identify and prioritize threats and risks by providing additional criteria to evaluate the potential impact to the organization. Traditionally, in risk assessment methodologies such as OCTAVE, the organization's evaluation criteria are used to identify those risks that need to be mitigated. Using CSFs to determine which risks to mitigate can enhance this process because it provides an explicit tie to the organization's business drivers. This can make up for potential errors caused by poorly developed evaluation criteria or a misapplication of the criteria.

Risk mitigation is a burden on the organization that must be considered within the context of the potential benefits (i.e., prevention of risk or reduction of impact) that can be achieved. By using CSFs as a guide, an additional and important variable can be considered in the cost-benefit analysis of risk mitigation strategies.

Figure 16 provides an example of affinity analysis between CSFs and risks that have been identified for critical assets. In this example, the organization is stating that the threat of alteration of "employee records" directly impacts the ability to "manage compliance." If this risk is realized, the "manage compliance" CSF will be impacted, and thus the risk should be mitigated.

Figure 16: Affinity Analysis for Determining Which Risks to Mitigate

¹ Henri Fayol's classic view of management includes the functions of planning, organizing, commanding, coordinating, and controlling. The effectiveness of each of these functions can be greatly enhanced if performed within the context of the organization's critical success factors. More information on Fayol's management functions can be found at http://www.onepine.info.

² More information on the OCTAVE method can be obtained from http://www.cert.org/octave.

³ The technique used to perform affinity analysis is provided in Appendix A."

⁴ Information assets represent the data and information, in either physical or electronic form, that is critical to the organization. Technical assets represent those assets that support the storage, transmission, and processing of data and information and therefore are important to transforming data and information for use by the organization. People can be an asset to the organization as well for similar reasons--they can be a primary way of storing, transporting, or processing data.

⁵ Security requirements in these categories are commonly applied only to information assets. Technical assets have security requirements as well, but are not often described in terms of confidentiality, integrity, or availability.

Appendix B: Case Study 1: Federal Government Agency

Introduction

This case study documents the use of the CSF method at a federal government agency of the United States. The CSF activity was undertaken as part of an information security risk assessment conducted by a team from the Networked Systems Survivability program at the SEI. The risk assessment focused on identifying the information security risks of a publicly available government Web service and developing a corresponding protection strategy to address those risks. The primary motivation for the CSF activity was to ensure that the assessment findings and the resulting protection strategy were aligned with the agency's organizational goals and objectives.

Acceptance of the CSF activity by the agency's management was not considered a criterion for success of the overall risk assessment. Instead, it was our belief that the CSF activity could be a catalyst for senior management support (of security activities) by characterizing IT security and risk findings in terms of business drivers. The agency's use of CSFs beyond this activity was always viewed as a secondary goal because, at the time, the use of the CSF method for the purpose of linking risk assessment and business drivers was still being researched and developed by the SEI team.

The Need for a CSF-Type Activity

Background

In 2002, we conducted an initial information security risk assessment for the federal agency presented in this case. The output of that assessment activity was a collection of information security vulnerabilities and risks to the public Web service and a suggested protection strategy. Since that time, a number of significant changes have occurred. For example, the immediate sponsorship, geographic location, contractor relationships and strategic partnerships, and architecture of the public Web service have changed. The agency also experienced a change in senior management, including the chief technology officer and a senior administrator. Similarly, some of the agency's operational units underwent changes not only in personnel but also in their placement in the organization's hierarchy. For example, the systems support and development personnel and their immediate supervisors were separated into functional teams instead of remaining in one organizational unit.

Because of these significant changes and the agency's previous assessment engagement, the SEI was asked to once again perform an information security risk assessment and to help the agency to develop a protection strategy. The assessment team began by conducting interviews with various staff members at the agency in order to get a better understanding of the current operational environment and to develop an appropriate scope. Immediately, it became apparent that there were a number of organizational disconnects. For example, the goals and objectives of senior management appeared to be significantly different from the goals and objectives of the operational staff. We also observed that the mission of the agency had changed since our last engagement and that all levels of the organization did not share a common view of the mission.

Positioning the CSF Activity

Considering these changes and challenges, it became evident that the assessment team and the agency would benefit from developing a set of CSFs. Thus, we initially proposed the CSF activity as a way to help us better understand the current direction of the agency and to ensure that the risk assessment was properly focused on those areas most important to the agency. However, the primary goal of the assessment was to identify security risks and to develop mitigating strategies. Because risk is highly relative to those assets and processes that are important to accomplishing the agency's mission, we also proposed that CSFs could help us prioritize and understand the agency's security risk in terms of how they may impede the accomplishment of CSFs. Thus, for these reasons, identifying CSFs became a logical first step to properly characterizing the current relationship between information security and the organization's business drivers.

Specific CSFs Derived for the Agency

This section describes the specific CSFs derived at the agency as well as the supporting themes and activity statements of each CSF. Activity statements for each CSF were logically organized against the general, technical, or customer management focus that the statement represented. Originally more than five CSFs and many more activity statements were derived for the agency. For the sake of brevity, consistency, and sanitization of the actual developed CSFs, this report only portrays a subsection of the original information to illustrate the type and nature of the content yielded through the activity.

The following information was broken into two subsections: first, the actual CSFs, supporting themes, and activity statements; and second, the threats and risks that exist or are perceived as impediments to the agency in meeting the CSFs.

Table 6: Agency CSFs

CSF 1--Manage the Technical Production Environment

Supporting Themes

Manage production operations to provide services and achieve customer requirements.

Manage risks to production environment and impacts to customers and business partners.

Activity Statements

General Goals and Objectives

Ensure that the production environment is capable of accommodating the anticipated requirements of customers, partners, and users.
Ensure that responsibility for the security and availability of the environment is accepted by the contractor.
Perform periodic stress testing to ensure system performance in the environment.

Customer-Specific Goals and Objectives

Ensure that heightened availability requirements are maintained for the Web service under conditions where availability is paramount.
Maintain performance requirements for the Web service.
Maintain the accuracy and integrity of Web service content with documented procedures.

Technical Goals and Objectives

Perform appropriate monitoring of the health of systems and information assets.
Patch and maintain software in the technical production environment on a regular basis.
Document, communicate, and test procedures for backup/restorative processes.

Other Related Themes

Technical/change management of the production environment is an essential function.
System monitoring services are important to overall information system health.

CSF 2--Provide High Value Customer Products and Services

Supporting Themes

Maintain the integrity and availability of the Web service.

Support cross-agency portals.

Support e-government initiatives and activities.

Promote products and services to a broader audience.

Activity Statements

General Goals and Objectives

Provide services and support to the other government initiatives.
Develop a thorough understanding of government initiatives in order to provide service and support.
Clearly articulate the agency vision, goals, and objectives to remain relevant among the growing e-government initiatives.

Customer-Specific Goals and Objectives

Ensure that Web service content is timely and relevant.
Maintain the appearance and availability of Web service services in accordance with the agency priorities.

Technical Goals and Objectives

Implement a content management system to support future growth.
Implement effective government-wide customer relationship management capabilities.
Implement profile management capabilities.

Other Related Themes

Web search services are an important component of providing the Web service functionality and for meeting the agency's mission.
Supporting cross-agency portals is a major future objective.

CSF 3--Develop and Manage Human Resources

Supporting Themes

Develop, train, prepare, and support employees to participate in the growth of the agency.

Manage partner human resources to achieve the common goals of the agency.

Activity Statements

General Goals and Objectives

Ensure that a capable, trained contractor staff is available to successfully meet contract requirements and objectives.
Ensure that the agency and agency personnel fully understand security requirements and convey them to contractors.
The skill and experience of subcontractors' personnel is critical to the successful delivery of security services to the agency.

Customer-Specific Goals and Objectives

Obtain senior management support for increasing the size of the Web service team to support additional objectives.
Ensure that time and local human resources can su

The Critical Success Factor Method: Establishing a Foundation for Enterprise Security Management

To the Reader

Acknowledgements

Introduction

1.1 Critical Success Factors

1.2 Enterprise Security Management

2 Background

2.1 Lessons from OCTAVE

2.2 Challenges for Security Management

2.3 Addressing Challenges with CSFs

3 History of the CSF Method

3.1 Beginnings

3.2 Evolution of the CSF Method

4 A CSF Primer1

4.1 CSFs Defined

4.2 Goals Versus CSFs

4.2.1 Relationship Between Goals and CSFs

4.2.2 Cardinality3 Between Goals and CSFs

4.2.3 The Superiority of CSFs Over Goals

4.3 Sources of CSFs

4.3.1 Industry CSFs

4.3.2 Competitive-Position or Peer CSFs

4.3.3 Environmental CSFs

4.3.4 Temporal CSFs

4.3.5 Management-Position CSFs

4.4 Dimensions of CSFs

4.4.1 Internal Versus External

4.4.2 Monitoring Versus Adapting

4.4.3 Importance of CSF Sources and Dimensions

4.5 Hierarchy of CSFs

4.5.1 Enterprise6 CSFs

4.5.1.1 Nature of Enterprise CSFs

4.5.2 Operational Unit CSFs

4.5.2.1 Nature of Operational Unit CSFs

4.5.3 Relationship Between Hierarchy and Source

4.5.4 Other Considerations

5 Applying CSFs

5.1 Historical Application of CSFs

5.2 General Advantages of a CSF-Based Approach

5.3 Using CSFs in a Security Context

5.3.1 Enterprise Security Management

5.3.1.1 Enterprise Security Management Defined

5.3.1.2 ESM and CSFs

5.3.2 Information Security Risk Assessment and Management

5.3.2.1 Determining Risk Assessment Scope

5.3.2.2 Selecting Critical Assets for Assessment

5.3.2.3 Identifying and Validating Security Requirements

5.3.2.4 Identifying Risks to Critical Assets

5.3.2.5 Setting Evaluation Criteria for Measuring Risk

5.3.2.6 Evaluating Threats and Mitigating Risk

Appendix B: Case Study 1: Federal Government Agency

Introduction

The Need for a CSF-Type Activity

Background

Positioning the CSF Activity

Specific CSFs Derived for the Agency

4 A CSF Primer¹

4.2.2 Cardinality³ Between Goals and CSFs

4.5.1 Enterprise⁶ CSFs