Software Engineering Institute | Carnegie Mellon University
Software Engineering Institute | Carnegie Mellon University

TSP Coach Exam Development

How the Certification Examination Was Developed: Certification

Examination Development Overview

Certification is the acknowledgement that an individual has attained a well-defined level of understanding or ability against a particular body of knowledge or skill set and has maintained that level of expertise over a specific period of time. Certification assesses skills that are relevant to a specific job role by validating the appropriate competency and skills needed by a candidate to enter the profession. To assess this level of understanding and ensure the competency of its practitioners, certification programs typically utilize an assessment instrument such as an examination.

Following is a summary of the key milestones that occurred in the development of the SEI-Certified TSP Coach examination including a job analysis study, question development by subject matter experts, and determination of the examination's pass point.

Job Analysis

Job analysis is the process of gathering, and analyzing information about a position in order to identify the position's tasks, functions, and roles, as well as identifying the competencies required to perform the work of the position. A job analysis may be used for multiple purposes. Some of the most common purposes are: determining professional entry requirements; selecting individuals for employment; developing employee training plans; and developing performance evaluation measures.

The key purpose in conducting a job analysis is twofold; to describe a job as it is actually performed and to understand the job well enough to reliably and accurately define worker requirements. When done well, a job analysis provides valuable position-specific information. For this reason, the SEI conducted a job analysis to lay the groundwork for the SEI-Certified TSP Coach program. To further guarantee the usefulness of the effort, the SEI selected the Ohio State University Center for Education and Employment, a recognized leader in this area, to perform the analysis.

Job analysis helps to create a shared view of a job, fostering greater acceptance among interested parties as to the job's actual description. It also provides documentation that allows the employer to chronicle and defend processes and decisions, should they be challenged.

The process for job analysis involves utilizing individuals with reputations for being the "top performers" at their jobs, working on a short-term team assignment with a qualified facilitator. Subject matter experts (SMEs) were recruited directly from government, industry, and the SEI's TSP Initiative Team. These individuals became the team of experts who collectively and cooperatively described the occupation in the language of the occupation.

Using the OSU CCC process, the team of experts worked under the guidance of a trained facilitator for two days to develop the job analysis research chart. The chart contains a list of general areas of competence called duties and specifies several tasks for each duty. Brainstorming techniques are used to obtain the collective expertise and consensus of the team of experts. The completed TSP Coach job task chart is a graphic profile of the duties and tasks performed by successful workers in the occupation and is available upon request.

The team of experts also identified the general knowledge and skills required of successful workers; the tools, equipment, supplies, and materials used; the important worker behaviors essential for success; and the future trends and concerns likely to cause job changes.

The next step is to statistically validate the duties and tasks findings through a community survey. Each competency area is measured in terms of frequency, importance, and ultimately, in criticality to the practice. Those competency areas that ranked highest in overall criticality measurement are migrated to the test specifications grouping for examination development.

Test specifications

The 1.5-hour SEI-Certified TSP Coach exam consists of 45 multiple choice questions. For each question, a set of answer options is provided, only one of which is correct. These questions were developed to gauge an individual's competency to be a TSP Coach. The exam content is derived from a combination of the competency areas developed from the TSP Coach job analysis study and requirements from the SEI TSP Certification team.

Test specifications include questions covering the following competencies:

  •  Competency Area 1: Prepare Organization for Initial TSP Usage
  •  Competency Area 2: Facilitate TSP Team Launch
  •  Competency Area 3: Guide Personal Process Usage
  •  Competency Area 4: Guide Team Process Usage
  •  Competency Area 5: Guide Management in TSP Usage
  •  Competency Area 6: Support Data Analysis Usage
  •  Competency Area 7: Support Organizational TSP Infrastructure

Question writing process

From the development of the test specifications to the determination of the exam's pass point, SMEs were involved in each phase of developing the SEI-Certified TSP Coach exam. The involvement was especially valuable in the question writing process. These experts were trained in the development of clear, non-trivial questions that test the knowledge required to perform as a TSP Coach.

Question Review Process

Upon completion of the question writing process, SMEs were once again called upon. At this point in the exam's development, the experts reviewed the questions for clarity, correctness, and appropriateness. In conducting such a review, questions were extensively critiqued, rewritten, and, if necessary, omitted.

Alpha Testing of Questions

The questions were alpha tested with both Master and Nonmaster-level alpha testers.  The results were analyzed and were used to identify questions that were problematic, including ones that needed to be removed because they exhibited poor discrimination properties.

Determining a Pass Point

The criterion-referenced cut score process used to determine the pass point for the SEI-Certified TSP Coach exam is one of the most commonly used methods of determining a pass point for licensure and certification exams. Once again, the input of SMEs was critical. By identifying the knowledge necessary to competently perform as a TSP Coach and estimating the difficulty that an individual possessing that knowledge would have in answering each question correctly, the SMEs arrived at the exam's pass point.

What to Study

Take advantage of a wealth of training resources that will complement the way you learn best. Everything from books to instructor-led training can help you enhance your knowledge and skills.