From June to December 2012, the Software Engineering Institute (SEI) at Carnegie Mellon University conducted a survey of cyber intelligence programs across the public and private sector. Known as the Cyber Intelligence Tradecraft Project (CITP), SEI researchers developed an indepth assessment process that exposed several shared challenges across organizations performing cyber intelligence. One of the most compelling problems was the dearth of training and education opportunities for analysts in the cyber intelligence field. The goal of the CITP was not to develop training and education offerings, but rather determine the state of the practice for cyber intelligence across multiple sectors. Part of this work included defining the core competencies and skills that make up a successful cyber intelligence analyst. Having analyzed the data from the CITP participants, the team first developed a mind map to illustrate these core competencies and skills. More detailed information can be found later in the paper.
The team reviewed existing course offerings and identified discrepancies between the ideal skill set for a cyber intelligence analyst and what courses are actually being offered. The team accomplished this by
This white paper begins by defining core competencies and associated skills that the CITP team has determined are necessary for a cyber intelligence analyst to possess. The paper then covers existing offerings for cyber intelligence analysis training and education and how they match to the skills necessary to work as a cyber intelligence analyst. Finally, the paper explores how the team conducted a gap analysis, and recommends some courses of action to address the current state of cyber intelligence analysis training and education.
In January 2013, the SEI held a workshop for organizations that participated in the CITP study. A portion of the workshop was devoted to eliciting specific skills and traits that organizations wanted from their cyber intelligence analysts. The team quickly discovered that there was no standard for what constitutes a cyber intelligence analyst. During the CITP study, participants were asked to define the skill they valued most in a cyber intelligence analyst. One organization responded, “five years of experience.” This anecdote is not an outlier in the data. Many organizations reported differing demands of cyber intelligence analysts, typically based on the size of the organization or the maturity of their cyber intelligence program. Often, organizations did not have clear expectations for what the analyst’s skills or competencies should be and decided that the way around this was to hire experienced analysts (typically from the government) and hope that the previous employer had sufficiently trained them, thus alleviating that burden from the hiring organization. While hiring away another organization’s analyst may solve an immediate need, it does little to address the crux of the problem. The implications of not having a standard set of skills and competencies for cyber intelligence analysts are three-fold.
First, as alluded to above, the absence of clearly defined competencies and skills presents organizations with hiring challenges. Generally, organizations have two options for staffing cyber intelligence analysts: take a non-technical analyst and provide them with training in cyber security, or take a technical practitioner and teach them to look at the bigger picture and analyze technical data through a strategic lens. While many CITP participants advocated for hiring an inquisitive, critical thinker with a liberal arts background, when we surveyed what their staff consisted of it was almost unanimously Certified Information Systems Security Professional (CISSP) or Certified Ethical Hacking (CEH) certified network security practitioners that were promoted out of help desk or incident response roles. The exceptions to these cases occurred in the government, where organizations have the resources and the time to send analysts through months of internal training programs that groom them to be cyber intelligence analysts.
Similarly, in the Air Force there is a 93-day program to train Airmen to become Network Intelligence Analysts2. Such opportunities are missing in the private sector, and many organizations cannot unilaterally afford to create and maintain a program like the Air Force program.
Second, the lack of competencies and skills resulted in inconsistent training plans, no training path at all, or a costly “grab bag” approach where analysts were sent to a mix of technical classes that often overlapped in content or failed to address non-technical skills expected of intelligence analysts. The SEI conducted a survey of more than 100 courses including programs and certifications offered by academic institutions and private industry, and more than 40 offered by various components within the Department of Defense (DoD). Some of these programs were advertised specifically as cyber intelligence courses, others were technical components of a larger program in intelligence studies. As the analysis outlined in this paper will show, these programs, with few exceptions, were ineffective at addressing all of the key skills necessary for a cyber intelligence analyst. Instead, analysts have to navigate a series of courses to develop the skills required for strategic analysis of technical data. Alternatively, our research suggests that analysts can learn much of these skills through on-the-job training with mentors, hands-on apprenticeships, and being exposed to real-world scenarios, data, and tools that are absent from many of today’s traditional classroom offerings.
Lastly, the lack of clearly defined skills and competencies for cyber intelligence analysts is a roadblock to professionalizing the workforce. In a community with such diverse backgrounds and experiences, it is difficult for the cyber intelligence community to establish standards that would benefit the profession. For example, government terminology is very different from the terminology used in industry. This discrepancy exists in part because the government has carefully shaped their definitions to allow the military and intelligence community to operate in cyberspace without breaking laws. For example, the definition of “computer network attack” was carefully crafted so that the activities of the intelligence community to gather data from targets would not constitute an “attack” (it is considered exploitation, not attack). However, the very same tactics are used by cyber actors against U.S. companies every day, and companies consider those activities as attacks against their networks. Standardizing the qualifications and training of cyber intelligence analysts would go a long way toward creating a standard lexicon and taxonomy. This standardization would improve the communication between analysts at different organizations, leading to improved collaboration on cyber threats. The challenge in cyber intelligence analysis is not a technology challenge – the technology exists. Structured Threat Information Expression (STIX) and the Trusted Automated Exchange of Indicator Information (TAXII) frameworks enable analysts are able to communicate and collaborate to produce useful analysis products. The main obstacle to effective cyber intelligence analysts is that the analysts themselves have different training, experiences, and backgrounds that prevent them from communicating effectively.
Although the team was thorough in their work, there were still limitations to the availability of data that could be collected. Limitations included the inability to collect every course offering available; searches were restricted to information found online, mainly course titles and descriptions. The team was able to collect information from a small number of government offerings. The team, however, did not review any classified courses. Phone conversations provided insightful information from both academia and private industry. In spite of these limitations, the team was able collect a large sample of courses, training, and certifications offered to cyber intelligence analysts. See appendix A Gap Analysis Spreadsheet.
Throughout the project the team was able to spend a significant amount of time with practitioners learning about the capabilities and personality types required to conduct effective cyber intelligence. This information was divided into two categories: traits and competencies. Personality traits are naturally ingrained in an analyst. These natural traits are often difficult to teach. However, mentors and educators can encourage the development of these traits. For example, a person may not be naturally persistent, but coursework requires them to continue to ask questions, find out more information, and become more thorough in their work.
Competencies are comprised of a set of teachable skills. Problem definition, scope management, and research methodologies are some examples of the skills that address the larger competency of critical thinking. The following graphics describe the essential core competencies, skills, and traits of the cyber intelligence workforce.
Critical thinking is essential for intelligence analysts. The foundation of successful intelligence work is the analyst’s ability to define the problem, apply research methods, and think strategically to recommend a course of action.
The analyst’s next step after defining the problem is to collect data. Data collection helps the analyst provide a decision maker with actionable recommendations. It is crucial for analysts to be disciplined when collecting and managing information and using that data to produce reliable recommendations.
Research Methodologies & Applications - Possess a working knowledge of diverse research methods and how to utilize methods to shape data gathering, analysis, and reporting. (This skill is also essential to the “Critical Thinking” core competency.)
Validation/Verification - Approach collected information with healthy skepticism and explore different possibilities to validate and verify findings and conclusions. (This skill is also related to the “Critical Thinking” core competency.)
Collection Management - Understand how to turn requirements for intelligence into collection requirements, collect, prioritize, and store information from multiple intelligence disciplines.
Open Source Data - Possess an operational understanding of how to ascertain, validate, and employ data from sources that are generally available to the public.
Defending Assessments - Have the ability to explain and defend the assessments and recommendations that are made.
“Strong communication skills” is such a common phrase that it is almost discounted when searching for an analyst. The team received overwhelming responses from skilled practitioners supporting the need for intelligence analysts to have the ability to communicate, write for leadership, and understand the audience.
Technical Writing - Provide sufficient technical detail to show data gathering and analysis to support recommendations.
Writing for Leadership - Understand how leadership will utilize recommendations and communicate relevant data.
Debating Skills - Explain and defend assessments and recommendations.
Knowing Your Audience - Understand the potential audience(s) of your product and craft the product to have the most impact to that audience.
Conflict Resolution - Know how to quickly identify a resolution to a discrepancy when collaborating on an intelligence project.
Attention to Detail - Show attention to detail when examining data and communicating recommendations.
Assimilate New Information - Take new information and be able to understand and utilize it throughout the intelligence lifecycle.
Public Speaking - Be able to present findings and recommendations in a public environment.
This competency outlines the basic computing concepts that analysts should utilize to understand the environment they are operating in and how to gather and use the data that they are collecting.
Networks & Networking - Understand and utilize physical components, types of networks, protocols, and topologies.
Operating-Systems - Understand and utilize different operating-systems and operating system components.
Databases - Understand and utilize database types and applications.
Programming - Understand and utilize programming processes, methodologies, and programming languages.
Scripting - Understand and utilize roles and types of scripting languages.
Data Mining - Understand how to pull information from large data sets and how to structure information for reuse.
Intelligence analysts should have an understanding of basic security concepts. This understanding should be used to collaborate and share information with an organization’s cybersecurity function.
Vulnerability Assessments - Identify and prioritize system vulnerabilities.
Cryptography - Utilize techniques and technologies for secure communications
Technical Architecture - Utilize a framework for the networks, Open Systems Interconnection (OSI) model.
Information Architecture - Understand categorization and organization of data, data access patterns, permissions, data flow, and governance.
Network Defense - Understand how to protect, monitor, and respond to network attacks, reconnaissance, and intrusions.
Incident Response - Utilize incident management process and response to computer events.
Intelligence analysts must be well versed in the techniques that actors utilize to attack an organization.
Malware - Understand intrusive and disruptive software–viruses, worms, Trojans, rootkits, etc.
Penetration Testing - Utilize simulated attacks to evaluate and improve network security.
Social Engineering - Understand that social engineering can be used to manipulate employees, partners, and suppliers into provide information or unknowingly perform malicious activities.
Web Servers - Understand security concerns associated with web content, physical and virtual web servers.
Wireless networks - Understand types of wireless networks, wireless devices, and vulnerabilities.
Web Applications - Understand how web applications can be exploited.
After defining the traits, competencies, and skills of an ideal intelligence analyst, the team set out to understand the coursework offered to current and prospective analysts. To determine the current training and education opportunities, the team surveyed more than 150 courses, trainings, and certifications offered by academic institutions, private industry, and the DoD.
From the collected data, the team learned that government, private industry, and academic institutions do address more than half of the competencies and skills needed to become a cyber intelligence analyst. The team also confirmed that there are several discrepancies between the skills that analysts need and training opportunities that are offered. The training paths to become a qualified cyber intelligence analyst are inconsistent or nonexistent in some cases. This puts the analyst at a disadvantage because they are not taught content that gives them both the technical and non-technical skills required to perform their job. This also puts hiring organizations (both government and industry) at a disadvantage because they need to invest time and money to appropriately equip their analysts. The team found that academic institutions offered a variety of programs that were either technical or non-technical in nature. Not one program offered an ideal mix of classes; there was frequent overlap in content or failure to address skills that are specific to intelligence analysts. Furthermore, academic institutions are not always able to provide courses that use relevant tools and technology with current data and threats; however, private organizations have the ability to offer these courses by exposing analysts to real-world scenarios, data, and tools. Analysts can also learn skills not taught in classrooms through on-the-job training and hands-on internships provided by an organization.
The team was only able to capture a small amount of data that pertained to the programs the government makes available to cyber intelligence analysts, while still keeping the review of coursework at the unclassified level. From the data the team collected, it was clear that the government offers programs and training that are very resource-intensive. Government offerings are more expensive and require longer time commitments than other training programs. Although government organizations that create and maintain their own training programs are relatively successful, it is not organizationally cost effective for private industry to create and maintain their own internal cyber intelligence training programs.
Surveys showed that organizations hire either a) analysts, training them in cyber security, or b) technical experts, teaching them to analyze technical data strategically. The team discovered that although organizations said they wanted to hire an inquisitive, critical thinker with a liberal arts background they were, in fact, mostly hiring technical experts. This approach to hiring staff does not work; organizations usually fall short when offering courses and training that focus on security and neglects analysis.
Academic institutions offer a wide selection of courses that range anywhere from basic information technology to information security. This broad range gives analysts options; however, it does not give them a clear path to follow when their focus is cyber intelligence. Academic institutions are great at producing individuals that understand analysis but their courses lack the relevant tools and data an analyst needs to develop “real-world” skills.
To identify the existing gaps between the current and desired training and education opportunities, the team defined the core competencies and the correlating skills that are necessary for a cyber intelligence analyst. Next, the existing training and education offerings were matched to the newly identified core competencies and skills, shown in the Appendix A Gap Analysis Spreadsheet. The analysis shows that although the current offerings do have pockets of excellence the current curriculums do not address 35 percent of desired skills, shown in Figure 1. The deficient skills within the Computing Fundamentals, Information Security, and Technical Exploitation competencies are all technical skills that are taught in academic institutions and by private industry organizations. However, in academic institutions these courses often are not offered to intelligence analysts. The deficient skills found within the Critical Thinking, Data Collection & Examination, and Communication & Collaboration competencies are less technical in nature and are often taught as components of Liberal Arts courses, rather than having its own dedicated course. For example, the ability to know your audience and pay attention to detail are skills that should be taught to any analyst in any course that involves report preparation, presentation, and analysis. Courses need to address multiple skills in a single offering, equipping the analyst for a successful cyber intelligence career. For example, our research identified a course from the Naval Post Graduate School, Cyber Systems and Operations Research Methods, that teaches research methodologies and applications, collection management, analysis, and writing.
The CITP uncovered deficiencies in training and education opportunities for cyber intelligence analysts. The project team, through surveys and research, determined the traits, competencies, and skills that should create the ubiquitous and clear standard for what constitutes a cyber intelligence analyst. The team offers the following courses of action to greatly improve the education, recruitment, and professional development of cyber intelligence.
Academic institutions, private industry, and government should use the competencies and skills from the Intel Competencies Mind Map to review the current skill set of their intelligence analysts, target specific skills that are in need of improvement, then seek courses that specifically address those skills. Alternatively, organizations can develop training that will address the skills and competencies needed to perform cyber intelligence. Finally, organizations should identify a clear learning path for analysts that consist of core competencies and skills identified.
Organizations should invest in the time to review and rewrite job descriptions for new employees. Using the competencies and skills mind map, organizations should identify which competencies and skills address their current needs, and then craft the job description to look for the best candidate. During the interview process use the list of traits to ask questions to determine if the candidate has the natural abilities required to be a successful intelligence analyst.
During the participant workshop, organizations emphasized the desire to establish internships and apprenticeships. The relationship between academic institutions and hiring organizations is mutually beneficial. The hiring organization is able to gain short-term talent, the ability to recruit qualified individuals, and provide feedback to schools. The academic institution is able to build a relationship with the hiring organization, which benefits the students, and will give them feedback to improve their curriculum.
Industry and academic institutions offer an array of cyber intelligence education and training opportunities through traditional classroom instruction, online tutorials, and certification programs. Despite the variety of offerings, project participants indicated that these programs do not fully meet their organization’s needs. Participants expressed that the use of traditional teaching methods for cyber intelligence is antiquated because they fail to produce analysts with the versatility and critical thinking skills needed to succeed in the diverse and ever-changing cyber environment. The team believes that many of the deficiencies in today’s cyber intelligence training programs can be addressed through advanced tradecraft technology. This technology leverages cyber intelligence, computer science, and visual analytics to provide a dynamic, virtual platform for individuals to hone technical and analytical skills. The principle goal of advanced tradecraft technology is to place students into real-world scenarios where the student is able to conduct multi-source analysis, understand critical information that is needed to make recommendations and decisions, and can impact strategic decisions.