Advancing AI for Mission Success

The Pentagon’s 2023 Data, Analytics, and Artificial Intelligence Adoption Strategy acknowledged that the challenges to adopting artificial intelligence (AI) and machine learning (ML) in defense programs go beyond the technical: “This strategy outlines our approach to improving the organizational environment within which our people can deploy data, analytics, and AI capabilities for enduring decision advantage.” Some of these organizational challenges include the lack of guidance around operator trust and tool selection, as well as workforce readiness. In 2025, the SEI developed several solutions to fill these gaps.

Establishing Standards to Advance Operator Trust

The Department of War’s (DoW) directive 3000.09 allows for the development of AI-enabled lethal autonomous weapon systems (LAWS). The absence of established standards for AI assurance makes it difficult for defense programs to assess the trustworthiness of LAWS before deployment. To advance trustworthiness in AI weapon systems and develop methods for evaluating operator trust, the Under Secretary of War for Research and Engineering tasked the SEI with piloting the Center for Calibrated Trust Measurement and Evaluation (CaTE) in 2023.

At the conclusion of the pilot in 2025, the SEI’s AI Division organized CaTE research into two resources for LAWS developers. Reference Architecture for Assuring Ethical Conduct in Lethal Autonomous Weapon Systems (LAWS) is a framework for creating systems that can embody and govern ethical principles through requirements and system design. CaTE Guidebook for the Development and TEVV of LAWS to Promote Trustworthiness provides operational test and evaluation (OT&E) and developmental test and evaluation personnel with observations and recommendations for effectively developing, testing, evaluating, verifying, and validating (TEVV) LAWS. See more resources for developing trustworthy autonomous systems.

Structuring Evaluation for MLOps Tools

Organizations seeking to build and adopt AI-enabled systems face the difficulty of identifying the right capabilities and tools to support Machine Learning Operations (MLOps) pipelines. For the many DoW software programs that are new to AI or not yet operating systems at scale, navigating the vast array of available tools can be particularly difficult.

To combat this challenge, SEI researchers Violet Turri and Emily Newman developed the MLOps Tool Evaluation Rubric to help acquisitions teams and engineering programs score and assess ML tools based on key criteria and metrics essential for successful AI and data initiatives.

Designed to standardize and enable MLOps acquisitions, the customizable rubric helps organizations pinpoint their priorities for MLOps tooling, evaluate key capabilities, and ultimately choose tools that will provide comprehensive support for ML developers and systems throughout the entire lifecycle, including data exploration, model deployment, and monitoring.

Uncovering Hidden Talent in the DoW Workforce

AI readiness remains a top priority for the DoW workforce. However, identifying skilled workers can be challenging when they lack traditional credentials. To address this challenge, the SEI’s AI Division partnered with the Department of the Air Force Chief Data and AI Office (DAF CDAO) to develop strategies for uncovering and assessing hidden workforce talent for data and AI roles.

The collaboration, led by the SEI’s Dominic Ross, produced a Data/AI Cyber Workforce Rubric (DACWR) for evaluating competencies identified within the DoW Cyberworkforce Framework, prototype work sample assessments that capture a data science pipeline (data processing and analysis, model creation, and reporting), and a proof-of-concept platform, SkillsGrowth, allowing workers to build profiles that highlight their expertise and assessment performance and enabling managers to identify the data and AI talent they need.

The SEI’s AI Division is now seeking mission partners to improve the rubric, assessments, and the SkillsGrowth platform. Together, these efforts will expand the DoW’s capacity to evaluate and support AI workforce readiness.

As the DoW shifts from inventor and primary investor in new technology to technology acquirer, the DoW has to be able to understand these technologies, trust them, and apply them in their domain in ways that they can count on.

Matt Gaston

Director, SEI Artificial Intelligence Division

Turning Research into Operational Readiness

Matt Gaston, director of the SEI AI Division, emphasizes the importance of reliable AI within the Pentagon. “As the DoW shifts from inventor and primary investor in new technology to technology acquirer, the department has to be able to understand these technologies, trust them, and apply them in its domain in ways that it can count on.”

By guiding the development of trustworthy AI systems, clear acquisition processes, and a stronger defense AI workforce, the SEI transforms cutting-edge research into assured operational capabilities.