Artificial Intelligence Robustness (AIR) Tool
Created January 2018 • Updated November 2026
Modern analytic methods like artificial intelligence (AI) and machine learning (ML) models provide powerful tools that have revolutionized prediction capabilities through their capacity to analyze and classify data.
The Department of War (DoW) and the federal government are accelerating their use of AI models to make classifications and predictions in operations, mission planning, simulation, logistics, and threat modeling while maintaining trust in AI robustness. The Memo on Accelerating Federal Use of AI through Innovation, Governance, and Public Trust, released by the Office of Management and Budget, offers guidance on federal use of AI and includes the need for additional monitoring activities.
The methods used by DoW use depend on correlations; however, such approaches fail to account for confounding in the data, which prevents accurate modeling of cause and effect and often leads to bias. Edge cases, drift in data/concept, and emerging phenomena undermine the significance of correlations relied upon by AI. New test and evaluation methods are therefore needed for ongoing evaluation. The AIR method and tool allow users to gauge AI and ML classifier performance with data-based confidence
Improving Confidence in AI Predictions
AIR helps increase confidence in AI outputs by answering fundamental questions about those outputs. For instance, can a classifier’s predictions be relied on in many different use cases, or just one? Are a classifier’s predictions accurate over time? AIR allows users to evaluate this data, understand causal relationships, and identify potential causes of bias. This provides insight into potentially spurious correlations throughout the model-building process.
AIR brings causal discovery, identification, and estimation together into one automated pipeline suitable for almost anyone seeking to evaluate a classifier – from non-experts to experienced operators. By employing a body of research that applies causal learning techniques to produce cause-effect graphs (causal discovery) and evaluate cause-effect relationships (causal inference), AIR can inform feature engineering to resolve bias and develop robust predictive models. It can also help users gauge AI and ML model performance to improve new and existing models with unprecedented confidence.
The AIR method and tool provide an evaluation capability for users to understand causal relationships in their data and identify potential causes for bias, providing useful insight into spurious correlations before, during, and after model building.
Linda Parker Gates - AIR/TAILOR Principal Investigator and Initiative Lead, Acquisition Policy, Practice, and Integration
Providing Causal Insights
The results of AIR analyses build a collection of confidence intervals for different cause and effect scenarios, creating a rich testing regimen that can inform model development or troubleshoot changes in model performance during cycles of training. The results of the AIR method include
- causal graphs to visualize the data relationships and determine if all needed features are included in the dataset
- identification of features likely to be causing bias in the prediction, to help inform better decisions when selecting training data representative for the scenario of interest
- confidence intervals for a cause’s average effect on the result, which can serve to validate model training
- comparison of the causally informed predictions with the model predictions in an already trained AI model; in these cases, the AIR tool can show when the AI model should not be trusted and suggest improvements
Software and Tools
AI Robustness (AIR) Tool
The AI Robustness (AIR) tool allows users to gauge AI/ML classifier performance with unprecedented confidence.
Learn MoreLearn More
Applying Causal Learning to Evaluate Large Language Models (LLMs)
•SEI Report
This report describes how the SEI applied causal discovery to LLM summarization, exemplifying a way to address bias when evaluating complex new technology.
ReadFrom Data to Performance: Understanding and Improving Your AI Model
•Podcast
In this podcast, SEI researchers discuss the AI Robustness (AIR) tool, which allows users to gauge AI and ML classifier performance with data-based confidence.
ListenMeasuring AI Accuracy with the AI Robustness (AIR) Tool
•Blog Post
Understanding your artificial intelligence (AI) system’s predictions can be challenging. In this post, SEI researchers discuss a new tool to help improve AI classifier performance.
READ