TSP Symposium 2014 Presentations

This file contains presentations from the Ninth Annual TSP Symposium on November 3-6, 2014 in Pittsburgh, Pennsylvania. 

Sessions and presentations included:

*Advanced Modeling of Teaming Data to Enable Superior Team Performance, Robert W. Stoddard, Dan Bennett, David Webb, Rushby Craig, and Lance Moore
*An Extension of the PSP PROBE Process to Help Students Make More Reliable Estimates in Early Stages of PSP Training, Yoshihiro Akiyama
*An Incremental Life-Cycle Assurance Strategy for Critical System Certification, Peter H. Feiler
*Architecture Best Practices for Project and Technical Leaders, Felix Bachmann, Jim McHale, and Timothy Morrow
*A Vendor Development Program: Smart Clients Are a Success Factor, Francisco Aleman
*A Zero-Depth Entry to Using TSP: How TSP Turned Around the Smart Grid Maturity Model Project, Julia L. Mullaney and Summer C. Fowler
*Case Study of Toyota Unintended Acceleration and Software Safety, Philip Koopman
*Common System and Software Testing Pitfalls, Donald Firesmith
*Evolving Postmortems as Teams Evolve Through TxP, Brad Hodgins
*Graphical Recordings of the TSP Symposium
*Information Flow: The Secret to Successful Teamwork, Jesse Schell
*Insider Threats in the Software Development Life Cycle, Daniel L. Costa and Randall F. Trzeciak
*Introduction to Software Product Lines, Patrick Donohoe
*Scrum: Creating Great Products and Critical Systems – What to Worry About, What’s Missing, and How to Fix It, Neil Potter 
*SEMPR: The TSP Software Engineering Measured Performance Repository, William Nichols and Yasutaka Shirai 
*Software Architecture Decision-Making Techniques, Elizabeth Correa
*Taking the Team Meeting to the Next Level, David Saint-Amand
*Tales from the Quality Journey, Darryl Davis 
*The ACE (Accurate Confident Estimating) Process, Carl Wyrwa
*The Executive View: Beyond the Methodology, David VanEpps
*The Impact of the PSP on Software Quality: Eliminating the Learning Effect Threat Through a Controlled Experiment, Fernanda Grazioli, Diego Vallespir, Silvana Moreno, and Leticia Perez-Queiruga
*TSP History and Evolution at Cadence Design Systems, Elias Fallon
*TSP in Non-implementation Phases: An Experience in How Disciplined Measurement Has Helped Overcome Obstacles in Deploying TSP in Mexican Organizations, Blanca Gil 
*TSP-PACE: Process and Capability Evaluation, an Experience Report, Antonio Mejorado, Rafael Salazar, and William Nichols
*Under N: Acceptance to Delivery in N Hours, Umashankar Velusamy
*Using PSP/TSP in a Performance Review, Larry Whitford and Nicole Flohr
*Wild, Wild West—How to Corral All Your Developers into Creating Secure Code, Jonathan Beck 





Advanced Modeling of Teaming Data to Enable Superior Team Performance, Robert W. Stoddard, Dan Bennett, David Webb, Rushby Craig, and Lance Moore

This TSP Symposium 2014 presentation describes periodic team survey data collection initiated at Hill Air Logistics Complex during 2014 that was analyzed and modeled using traditional statistical analysis and modern structural equation modeling (SEM). Results of the analysis and SEM will demonstrate how team performance may be enhanced through a focus on key team strengths and weaknesses associated with team practices discussed in TSP: Leading a Development Team, by Watts Humphrey. The authors seek to show the practicality of collecting such practice data and combining it with existing TSP data to reduce barriers to successfully implementing stronger team practices and achieving more sustainable team performance.





An Extension of the PSP PROBE Process to Help Students Make More Reliable Estimates in Early Stages of PSP Training, Yoshihiro Akiyama

Students experience significant over- or underestimates in new program development when it follows the Personal Software Process (PSP) PROBE process. This TSP Symposium 2014 presentation overviews how this potential problem can be reduced or solved by extending the PSP PROBE process.

A strategy is to find a logical evaluation combination of the productivity, correlation, and β1 and β0 used in the PROBE process. The estimated average size and time for a new program to be developed are obtained when method C is selected. The estimated proxy size is obtained for the new program through the PSP relative size table applied to the conceptual design. The estimated average size replaces the estimated proxy size when a student selects method C. If the estimated average size is not reasonably consistent with the past productivity data, the student should select method D for the size estimate. The similar discussion holds for the time estimate. 

Another extension that is effective in the PSP PROBE process is to allow selecting method A or B when high correlations are recognized between the estimate and actual values, regardless of whether β0 or β1 satisfies the desired conditions respectively described in the PSP PROBE process, although the PROBE process suggests selecting method C for these cases. 

After this extended PROBE process was presented, PSP students demonstrated reaching fewer over- or underestimates on the size and time of new programs.






An Incremental Life-Cycle Assurance Strategy for Critical System Certification, Peter H. Feiler

This TSP Symposium 2014 presentation describes an architecture-led incremental assurance strategy throughout the development life cycle to address the challenges of certifying mission- and safety-critical systems that have become increasingly software reliant. This strategy is pursued in an international SEI, industry, and government collaboration. For aircraft, software as percentage of total system cost has grown from 33% in 1997 to 67% in 2010, with verification-related software rework cost alone exceeding 50%. Systems are currently verified against ambiguous, incomplete, and inconsistent requirements. Industry studies show that 70% of embedded software system defects are introduced in requirements and architecture design, while 80% are discovered post-unit test, with rework cost as much as 300–1,000 times the cost of in-phase correction.


The strategy involves a paradigm shift from build-then-test to an architecture-centric engineering approach that utilizes analytical virtual system integration based on the SAE Architectural Analysis & Design Language standard to discover problems earlier in the life cycle. This paradigm shift is being pursued by an international aerospace industry initiative known as System Architecture Virtual Integration, with return on investment studies showing major cost savings.

The strategy measurably improves requirement coverage through architecture-led requirement specification—incorporating operational requirements such as performance, timing, safety, reliability, and security—and systematically addressing hazards in the process. The strategy applies contract-based compositional verification one architecture layer at a time to ensure that requirements are addressed throughout the life cycle. Finally, the strategy incrementally manages an assurance plan and its execution throughout the life cycle, producing assurance case artifacts for certification.






Architecture Best Practices for Project and Technical Leaders, Felix Bachmann, Jim McHale, and Timothy Morrow

This TSP Symposium 2014 presentation explains that architecture development is one of the earliest, key identifiers of potential risks in a project's development life cycle. Architecture provides the foundation for the project's entire life cycle and is used to help address the important areas of a project such as design, schedule, estimates, testing, users' satisfaction, team structure, and training. TSP makes use of and expands on these areas. For example, an architecture development process that includes estimating architecture development effort based on quality attribute scenarios fits nicely into the initial conceptual design efforts. Taking an architecture-informed approach and inferring its implications and impacts for a project's development success is a key skill that leaders need in order to identify risks at the start of a project or determine a baseline for an existing project. This presentation describes a set of architecture best practices based on experiences working with commercial and government departments and agencies in the areas of software, system, enterprise architecture, and system-of-systems development. 

The architecture best practices focus on four areas: visual representations, specifications, processes, and acquisition. In these four areas, two levels of supporting artifacts are identified and integrated in a timeline to help a leader plan for a new project, support assessing a project's current health, and develop a baseline from which to identify areas for process improvement. The first level provides a minimalist framework that will work for a project of any size to provide an architecture-informed snapshot. The second level is more comprehensive and will lend itself better to larger projects.





A Vendor Development Program: Smart Clients Are a Success Factor, Francisco Aleman

In this 2014 TSP Symposium presentation, the Performance and Capability Evaluation (PACE) establishes a framework to build the management capabilities that transform outsourcing companies into strategic business partners. Firms with an outsourcing strategy need data for decision making and need the ability to govern the service portfolio in order to enjoy the benefits of successful relationships with software providers. Internal resources’ skills correlate directly to the firm’s transformation success. To obtain this success, the client must help its provider succeed by helping it grow and improve capabilities and performance.

The PACE framework offers measurable goals, a baseline for process improvement, and a maturity path to documentation of metrics and performance reporting. This framework can support a Vendor Development Program based on the four dimensions of PACE: coverage, fidelity, performance, and customer satisfaction. It also provides the database structure to collect the vendors’ project data. Establishing this program poses a double challenge: it is not only one firm’s internal transformation, it also requires external firms’ transformation.






A Zero-Depth Entry to Using TSP: How TSP Turned Around the Smart Grid Maturity Model Project, Julia L. Mullaney and Summer C. Fowler

This 2014 TSP Symposium presentation describes how basic Team Software Process (TSP) principles were used to bring the Smart Grid Maturity Model (SGMM) project under control. The SGMM project is a non-software project composed of geographically dispersed, part-time resources from multiple departments within the organization. This presentation follows the journey of the SGMM from conducting an initial project launch, through developing two version releases of the SGMM product suite. We discuss how the team first got its schedule under control, then focused on quality and cost. We pay particular attention to how the MS Project was used to budget and track project costs, including manpower, travel, subcontractors, and all other expenses. We also discuss how other aspects of the TSP were tailored to our situation, including roles, lessons learned, and weaknesses with our approach.






Case Study of Toyota Unintended Acceleration and Software Safety, Philip Koopman

Investigations into potential causes of Unintended Acceleration (UA) for Toyota vehicles have made news several times in the past few years. Some blame has been placed on floor mats and sticky throttle pedals. But a jury trial verdict found that defects in Toyota's Electronic Throttle Control System software and safety architecture caused a fatal mishap. This verdict was based in part on a wide variety of computer hardware and software issues. In this TSP Symposium 2014 keynote presentation, Philip Koopman outlines key events in the still-ongoing Toyota UA story and pulls together the technical issues that have been discovered by NASA and other experts. The results paint a picture that should inform not only future designers of safety-critical software for automobiles but also all computer-based system designers.





Common System and Software Testing Pitfalls, Donald Firesmith

This TSP Symposium 2014 presentation explains that projects, testing programs, and testers often fall into commonly occurring testing pitfalls that make testing unnecessarily less effective at uncovering defects, less efficient in terms of testing resources, and more frustrating to testing stakeholders. Based on the experience of many testers, we have developed a taxonomy of 145 common testing pitfalls organized into 21 categories. Each pitfall has been analyzed and documented in terms of title, identifier, description, potential applicability, characteristic symptoms, potential negative consequences, potential causes, recommendations, and related pitfalls. The first 92 pitfalls were documented in the book Common System and Software Testing Pitfalls: How to Prevent and Mitigate Them. Since the book manuscript was baselined for publication, a further 35 pitfalls and four pitfall categories have been identified, analyzed, and documented.





Evolving Postmortems as Teams Evolve Through TxP, Brad Hodgins

In working with TPI (Team Process Integration) teams in NAVAIR, patterns are beginning to emerge concerning which topics to focus on when performing postmortems. Showing a team how it has improved in some aspect is a very strong motivating factor to help the team embrace process improvement and to show how easily improvement can be achieved. First postmortems, therefore, are important in sustaining a team’s interest in process improvement. As a team evolves and begins to use more complex plans, more extensive postmortems become possible. We have developed a standard set of postmortem topics that can be introduced piecewise as a team gains experience and that can be analyzed regardless of that team’s domain. Standardization allows organizational managers to become familiar with the charts presented in the postmortem out-briefs, which, in turn, show management that the team is improving its performance in some aspect, regardless of whether the team is well on its way to becoming a TxP (Team-X-Process) team or just starting the TPI. The presentation will walk through the graphs that have been most effective in presenting analysis results at the various levels of evolution as a team works toward becoming a TxP team. These analyses improve planning, improve communication with management, and sustain team interest in process improvement.





Graphical Recordings of the TSP Symposium

A graphic designer captured nine presentations from the TSP Symposium 2014 in graphical recordings.




Information Flow: The Secret to Successful Teamwork, Jesse Schell

Video-game development is arguably one of the most difficult kinds of software development due to its combination of performance requirements, diverse team, immovable deadlines, and the fact that the software must be more than functional, it must also be fun. In this TSP Symposium 2014 keynote presentation, Jesse Schell explains what he has learned from 20 years of professional game development about the secret forces that help teams succeed or cause them to fail.




Insider Threats in the Software Development Life Cycle, Daniel L. Costa and Randall F. Trzeciak

This TSP Symposium presentation explains that the software development life cycle presents a wide array of attack vectors for malicious insiders. The software produced, and its associated artifacts, are assets that an organization must protect. The data collected by or entered into software can be the target of theft, tampering, and other types of malicious activity. The business processes automated by software can be severely impacted when software is faulty or services are unavailable. Through the CERT Division's insider threat research, we have collected numerous cases in which insiders exploited vulnerabilities in software development processes to cause harm to their organizations. In this presentation, we discuss patterns and trends in these cases, focusing on similarities in attack techniques, targets, and motivations. We also present mitigation strategies for commonly exploited vulnerabilities and make the case for the creation of a secure software development process as a critical piece of a robust insider threat program.





Introduction to Software Product Lines, Patrick Donohoe

This TSP Symposium 2014 presentation explains that a software product line is a set of software-intensive systems sharing a common, managed set of features that satisfy the specific needs of a particular market segment or mission and that are developed from a common set of core assets in a prescribed way. Organizations developing a portfolio of products as a software product line are experiencing order-of-magnitude improvements in cost, time to market, staff productivity, and quality of the deployed products.


This presentation will introduce the essential activities and underlying practice areas of software product line development. It will review the basic concepts of software product lines, discuss the costs and benefits of product line adoption, introduce the SEI Framework for Software Product Line Practice(SM) guidelines, and describe approaches to applying the practices of the framework.





Scrum: Creating Great Products and Critical Systems – What to Worry About, What’s Missing, and How to Fix It, Neil Potter 

This TSP Symposium 2014 presentation explains that the interest in Agile/Scrum software development practices continues to grow as companies seek more efficient methods of developing software while meeting market demands for delivery. Scrum is a software development methodology based on Agile principles. Agile methodologies promote a project management process that encourages frequent inspection and adaptation; a leadership philosophy using teamwork, self-organization, and accountability; and strong customer involvement.

However, Scrum/Agile implementations don't always go as planned, and without some due diligence, chaos is easy. In this session, Neil enumerates the problems to look out for and provides example corrective actions:

Scrum/Agile overview
What to use from the Agile Manifesto
Definition of Scrum
Scrum risks to look out for and what to do about them:
Mistaking speed for progress
One-liner requirements (the devil in the details)
Missing architecture/design
Missing final system test/validation
Missing configuration management
Missing risk management
Missing process assurance (known as ScrumBut)





SEMPR: The TSP Software Engineering Measured Performance Repository, William Nichols and Yasutaka Shirai

In 2013, SEI presented the first summary of process and projects statistics submitted by Team Software Process teams to the SEI. Since then, the more recent data using the Process Dashboard tool has been collected into a data warehouse provided by Tuma Solutions. This TSP Symposium 2014 presentation introduces data from this warehouse to the community and provides some initial benchmark statistics for use in project planning. Similarities and differences in how the data presented in 2013 are stored and retrieved are discussed. We also describe and present several “Fact Sheet” summaries that will be used to support some specific types of benchmarking and analysis.




Software Architecture Decision-Making Techniques, Elizabeth Correa

This TSP Symposium 2014 presentation explains that many architects and leads suffer from analysis paralysis. This session gives them a proven set of techniques to apply to all their software design decisions and build their confidence in their design decisions. Software architecture and design decision making falls under the category of “Wicked Problems.” There are no right answers. This session covers three aspects of making decisions to help the audience go back to their teams and make better decisions. The aspects covered include philosophies, actions, and psychologies of decision-making techniques.





Taking the Team Meeting to the Next Level, David Saint-Amand

This TSP Symposium 2014 paper reviews the development of both an effective and efficient tool for encouraging participation and capturing information at team status meetings: a meeting agenda presentation-framework in which the participants take personal ownership of the material for which they are responsible. The tool is effective because it helps teams touch on all the necessary points of data and communication, and it is efficient because it helps teams do this in a short amount of time and in a repeatable way.




Tales from the Quality Journey, Darryl Davis

This TSP Symposium 2014 presentation shares quality management experiences and resulting insights from a career of over 31 years in product development, manufacturing, management, and process improvement. We share a selection of stories from the first year to the present, such as

applying what we were taught
our first great team
producing near-zero-defect products
the ups and downs of tools and automation
design for the long term
simpler, smaller, better
losing things
the well-engineered, low-defect products that no one wanted
filtering out the defects
making reviews effective
preventing defects
We summarize these experiences and insights with our view of the required framework for achieving the highest product quality. Our experiences demonstrate that the best quality and performance are the result of the effective application across the organization of a collection of key principles and practices that are very broad in scope.

The stories and framework touch on a wide range of quality-related areas and practices, including leadership, teamwork, product management, customer needs and requirements, architecture and design, implementation, testing, configuration management, measurement, reviews, quality control and improvement, culture transformation, and customer support. We also relate our experiences, insights, and framework to guidance we have used over the years from our software engineering education, Total Quality Management, the Capability Maturity Model Integration, the Personal Software Process and Team Software Process, and Agile practices.






The ACE (Accurate Confident Estimating) Process, Carl Wyrwa

The ACE Process was established to provide a structured approach to developing estimates for teams that are striving to make Accurate and Confident Estimates. It begins with preparation activities and leveraging existing data prior to a planning session, describes specific activities applied during the planning session, and defines key finalization activities. It includes the introduction of some Agile (Scrum) elements that have been proven to promote full participation at all times using some enjoyable and effective techniques.





The Executive View: Beyond the Methodology, David VanEpps

On my first day as the executive of a large delivery organization, both my client and my manager described the challenge ahead by describing our past performance: abysmal. Poor quality, late projects, budgetary overruns, and an unprofitable business model. We were under intense pressure to improve fast, and we needed to make radical changes quickly and discontinuously.

We leveraged tools from the CMMI methodology and Team Software Process training to fight this battle, but we didn't start on a journey to seek certification. While that was a result of our journey, our approach went beyond the methodology to drive superior performance. We focused on using these powerful tools to solve critical business problems. 

We created a Pareto of everything that was wrong with our development organization. We focused on the biggest problems until we hit the 20% of problems that represented 80% of our pain. As we solved the first bar of the Pareto, quality, we also made huge dents in the second bar, on-time delivery. By the time we hit our 20% marker, we had addressed several other problems as well. First Time Quality improved by 66%, on-time delivery improved by 30%, budgetary overruns dropped by 37%, attrition dropped by over 30%, and profit increased by 12%. 

Our client and our management acknowledged these improvements, for a minute, and then quickly turned their attention to the next 10 problems. I made a realization—we live in a world of Paretos. Your client, your leadership, and your team will always have a Pareto of what needs improvement. As an exceptional leader, you need to be aware of the Pareto and constantly manage the 20%. That is, are you dealing with imperatives (e.g., cost, schedule, quality, attrition), or are you dealing with nice-to-haves (flexibility, thought leadership, more speed, more profit)? All are equally important from your client's perspective, but from a software-engineering excellence perspective, fixing the imperatives is a survival skill; fixing the nice-to-haves will set you apart.





The Impact of the PSP on Software Quality: Eliminating the Learning Effect Threat Through a Controlled Experiment, Fernanda Grazioli, Diego Vallespir, Silvana Moreno, and Leticia Perez-Queiruga

This TSP Symposium 2014 presentation discusses how data from the Personal Software Process (PSP) courses indicates that the PSP improves the quality of the developed programs. However, since the programs (exercises of the course) are in the same application domain, the improvement could be due to programming repetition. In this research, we try to eliminate this threat to validity in order to confirm that the quality improvement is due to the PSP. In a previous study, we designed and performed a controlled experiment with software engineering undergraduate students at the Universidad de la República. The students performed the same exercises of the PSP course but without applying the PSP techniques. Here we present a replication of this experiment. The results indicate that the PSP and not programming repetition is the most plausible cause of the important software quality improvements.




TSP History and Evolution at Cadence Design Systems, Elias Fallon

This TSP Symposium 2014 presentation describes how Cadence has been deploying TSP to its R&D organization since 2011. With our large R&D organization of more than 3000 engineers, which is geographically, organizationally, and technologically diverse, we found many challenges in achieving fast, widespread adoption. In 2013, we applied the basic principles of TSP to our TSP deployment effort and developed an alternative approach. That approach, named Cadence Development Engineer Training (CaDET), was developed in the second half of 2013 and early 2014 and has been under active deployment since March 2014. Here we present the history of our TSP deployment, the challenges that led us to evolve our deployment methodology, and the results of that evolution: CaDET.





TSP in Non-implementation Phases: An Experience in How Disciplined Measurement Has Helped Overcome Obstacles in Deploying TSP in Mexican Organizations, Blanca Gil 

This TSP Symposium 2014 presentation describes Performance and Capability Evaluations in Mexico. Many of the companies that have had a successful PACE (Performance and Capability Evaluation) have been Mexican. And several of these companies not only have implemented TSP in software development processes but also have done it in processes that are key to achieving successful projects and high-quality products. We're talking about processes like Requirements Analysis, High-Level Design, and Solution Architecture.

These first TSP implementations in processes that do not develop software shared the following experiences:

They customized planning and quality metrics and committed to follow disciplined measurement activities, even when traditional tools and repositories were adjusted.
On average, these organizations could have been defined as small and medium enterprises. But even as very small entities, these projects were based on self-directed teams committed to challenging and achievable goals and encouraged a successful cross-training that team members consciously followed.
All the companies implemented TSP cycles for the first time in their organizations. Some started with non-development software projects in which virtually all productive roles in the organization (analysts, architects, developers, testers) participated, which substantially reduced the resistance to change and motivated senior executives to continue the TSP implementation in subsequent cycles of the projects.
This talk shares the challenges and solutions that we faced in these cases and how the circumstances of these projects reinforced the path toward cultural change promoted by TSP in the organizations.





TSP-PACE: Process and Capability Evaluation, an Experience Report, Antonio Mejorado, Rafael Salazar, and William Nichols

This TSP Symposium 2014 presentation describes how Tecnológico de Monterrey and the Software Engineering Institute began the Team Software Process (TSP) organization evaluation process in 2008 and completed initial pilots of the Process and Capability Evaluation (PACE) in 2013. The effort has been partially funded by the Mexican government not only in order to assure that Mexican companies implement the TSP effectively but also to have national statistics to demonstrate the improving status of Mexican industry. Continuing with the TSP-PACE, we have evaluated more than 10 organizations in Mexico, using a combination of TSP data, project records, and on-site review. We present some results of the evaluations conducted and describe how the evaluations support both organizational performance improvement and national benchmarking. This presentation will discuss different approaches from the companies to meet their goals.




Under N: Acceptance to Delivery in N Hours, Umashankar Velusamy

How about delivering a business need from acceptance to production in less than 12 hours? Or delivering in the absolute time it would actually take—not an hour more and not an hour less? Under-N methodology provides a framework to uncover hidden capabilities within IT applications and IT application teams and then to use the capabilities to deliver a business need, change, or want in under N hours, where N is the absolute time it takes to deliver. It also elicits capabilities that may not yet be present but that are possible, while outlining four atomic change capabilities that, when implemented, will enable IT applications to deliver changes in an under-N fashion by mixing and matching. The Under-N Framework relies on strong collaboration, planning, and tight teamwork. The presentation outlines a template to frame these capabilities by asking the right questions to procure all the answers that will be needed to execute the capability and to engage the correct stakeholders. In addition, the presenter proposes ways to scale the capabilities across an enterprise by means of a challenge process and by celebrating the best.




Using PSP/TSP in a Performance Review, Larry Whitford and Nicole Flohr

Beckman Coulter has been using Team Software Process (TSP) since 2009, and today we have multiple TSP projects. This TSP Symposium 2014 presentation explores the use of Personal Software Process (PSP)/TSP data in the developers’ and testers’ performance reviews on two TSP projects. These two projects include 15 developers and 7 testers total. The projects include both new and maintenance development. Beckman Coulter uses a typical performance review process with goal setting in Q1 of the year, midterm reviews with managers, and final reviews with ratings at year’s end. Goals are weighted and ratings are 1 through 5, with 5 being the highest rating. The use of PSP/TSP in performance reviews for the two projects started informally in 2013 by holding developers and testers responsible for their schedule planning and general use of PSP data. There were no specific performance goals, however, and individuals were rated with subjective input from their functional manager. It was decided to be more formal for the 2014 performance year, and the manager developed specific goals based on the data and objectives of PSP/TSP. Individuals were brought into the process for goal setting, resulting in goals that strategically focused on both individual and team improvements. The 2014 performance goal objectives are reviewed and discussed along with the results to date.




Wild, Wild West—How to Corral All Your Developers into Creating Secure Code, Jonathan Beck

This TSP Symposium 2014 keynote talk discusses the many challenges facing chief information security officers when they are creating and maintaining the discipline necessary to ensure that their organizations implement secure coding practices. The options of using in-house developers, bringing in applications, and outsourcing to third- or fourth-party development teams expand the due diligence required to maintain a strong security posture.