Resources for Developing Trustworthy Autonomous Systems
The materials below comprise the set of resources prepared by the SEI and its collaborators to establish guidelines and standards that improve human understandability of artificial intelligence (AI) systems and promote the development of systems that are trustworthy and trust-promoting.
CaTE—Guidebook for the Development and TEVV of LAWS to Promote Trustworthiness
This guidebook provides operational test and evaluation (OT&E) and developmental test and evaluation (DT&E) personnel with observations and recommendations for effectively developing, testing, evaluating, verifying, and validating (TEVV) lethal autonomous weapons systems (LAWS) that function with machine learning (ML) models. The guidebook specifically addresses system trustworthiness and operator trust.
Reference Architecture for Assuring Ethical Conduct in LAWS
This reference architecture provides a framework for creating systems that embody and govern ethical principles through requirements and system design and ensuring these principles remain observable during testing and operation. The architecture also supports development processes that integrate ethical considerations throughout the entire development and sustainment lifecycles of autonomous systems.
A Guide to Failure in Machine Learning: Reliability and Robustness from Foundations to Practice
This guide provides practitioners with comprehensive understanding of machine-learning model failures. It categorizes the causes of failure as arising out of either a lack of reliability or a lack of robustness. The work offers formal definitions of failure modes from first principles and connects these concepts to engineering practices and real-world deployment scenarios.
CaTE Data Curation for Trustworthy AI
This report provides practical guidance about how to promote trustworthiness during the data curation phase of development for teams designing AI-enabled systems. The report defines key concepts and describes a series of steps that development teams, particularly data scientists, can take to build trustworthy AI systems through systematic data-curation processes.
Domain Knowledge Elicitation for Data Curation to Promote Trustworthiness in Artificial Intelligence
This paper offers a framework and question set for eliciting actionable domain knowledge from various sources, including field experts, to enable purposeful data curation that prioritizes performance in deployed environments. The work demonstrates how proper domain knowledge elicitation early in the AI lifecycle can significantly enhance system trustworthiness.
Human-Centric, Teaming-Focused Approach for Design and Development of Non-Deterministic Systems: A Human-Machine Teaming Design Framework
This proof-of-concept design framework provides a method to enhance the design of non-deterministic systems for effective human teaming. The framework complements existing design and development processes by focusing on enabling partnership and collaboration through design elements that foster willingness to work with non-deterministic systems over time.
CaTE Uncertainty Estimation: Decision-Driven Methodology for Designing Uncertainty-Aware AI Self-Assessment
This report presents a structured methodology for designing AI systems with integrated uncertainty-aware, self-assessment capabilities. The methodology enhances AI transparency, robustness, and trustworthiness by enabling models to evaluate and communicate prediction reliability through categorized self-assessment techniques and application-specific frameworks.
Decision-Driven Calibration for Cost-Sensitive Uncertainty Quantification
This work demonstrates an alternative uncertainty quantification calibration method that explicitly minimizes downstream decision costs rather than simply matching prediction confidence to data distribution statistics. The decision-driven approach directly considers the influence of uncertainty quantification on subsequent actions by decision makers.
HMT Studies
This work provides an overview of methodologies and instruments for eliciting user requirements and evaluating user trust in human-machine teaming scenarios. The studies covered in this document offer systematic approaches for understanding and measuring trust dynamics in collaborative human-AI systems.
The Myth of Machine Learning Non-Reproducibility and Randomness for Acquisition and TEVV
This SEI blog explains the prevalence of unpredictability in machine-learning systems, methods for addressing it, and the associated trade-offs. The work emphasizes the importance of expecting predictable and reproducible modes for machine-learning components, particularly for TEVV applications.
Mutation-Based Safety Testing for Autonomy: MOBSTA
MOBSTA is a robustness testing tool for ROS that tests autonomous systems under realistic but rare conditions that are not represented in training data and difficult to replicate during field testing. MOBSTA enables comprehensive safety evaluation by simulating challenging scenarios that would be dangerous or expensive to test in real environments.
MatchnMetric
This CaTE-developed metric and algorithm provides system-level robustness for ML-enabled automatic target recognition (ATR) systems. The technique uses labeled data to simulate perfect tracker operation and evaluate performance, resulting in more robust and trustworthy detection and tracking capabilities.