Software Engineering Institute

Major Department of Defense (DoD) acquisition programs put their defense systems through rigorous operational testing and evaluation (OT&E). The Office of the Director, Operational Test and Evaluation (DOT&E) for the Office of the Secretary of Defense (OSD) oversees this process. OT&E of both hardware and software has traditionally occurred near the end of the system development lifecycle.

However, the DoD Software Modernization Strategy calls for the delivery of “resilient software capability at the speed of relevance.” To meet DoD mission goals, DOT&E is compressing its activities and shifting them earlier in the cycle. Since 2020, DOT&E has engaged the SEI for help meeting these challenges of modern software OT&E.

To deliver DoD software quickly in a rapidly changing threat environment, DOT&E is adopting a wide range of development approaches and new technologies. Model-based systems engineering, digital twins, Agile, DevSecOps, simulation, artificial intelligence, machine learning, and virtual reality all can play a part in rapid, iterative development. At the same time, they raise questions for testing and validation.

“Take, for example, integrating into an Agile workflow,” said Nanette Brown, the SEI’s project lead on the DOT&E engagement. “The development team will do more validation with the customers of the specific capability being developed. But you must balance that with operational testing, which validates for the larger customer population.”

A team consisting of the SEI, other federally funded research and development centers, academic institutes, and DoD offices was assembled to assist DOT&E implement its strategy. The SEI helped develop policy and guidance documents for aligning software and cybersecurity T&E with modern software development practices such as Agile and DevSecOps. In particular, the SEI researched acceptance test driven development (ATDD) as a way to ensure that Agile processes capture OT&E requirements. This methodology will help DOT&E shift their activities to the left and reduce the likelihood of validation errors late in the lifecycle.

In 2022, the SEI began working with DOT&E to integrate its operational testing into the DevSecOps workflows of DoD systems development and acquisition programs. DOT&E is exploring the use of the SEI's DevSecOps Platform-Independent Model, which applies model-based systems engineering principles to DevSecOps pipelines, to guide this integration.

Another early-stage SEI effort is investigating the use of a continuous software bill of materials (SBOM). Using open source software provides tremendous advantages to DoD acquisition programs: low cost, high reliability, and flexibility. However, the open nature of the software also increases its supply chain risk. The information contained in SBOMs will provide DOT&E the opportunity to manage and monitor supply chain risk and ensure the safety and integrity of open source software.

Testing and evaluating complex, software-driven national defense systems require breadth and depth of knowledge. The engagement with DOT&E exemplifies how the SEI weaves various teams and projects to address DoD challenges.

These branches of work share one goal for DOT&E: to test and evaluate defense systems with the quickness of DevSecOps and the reliability of traditional processes. Having oversight, accountability, and confidence in a rapid and iterative development environment will allow DOT&E to deliver software at the speed of relevance.