Assuring Increasingly Autonomous Cyber-Physical Systems

2022 Year in Review

Assuring Increasingly Autonomous Cyber-Physical Systems

The 2018 Department of Defense (DoD) Artificial Intelligence (AI) Strategy touts AI as a potentially game-changing opportunity to improve its autonomous systems. The integration of AI has significantly enhanced the capability of cyber-physical systems (CPSs) to provide advanced control, situational awareness, and autonomy. These advances also make it harder to assure systems. Understanding the impact of AI functions on system safety requires a fine understanding of system architecture. In the Safety Analysis and Fault Detection Isolation and Recovery (SAFIR) project, SEI researchers are collaborating with academia to develop tools and a body of knowledge to analyze the impact of AI functions on the assurance of safety-critical systems.

The growing complexity of CPSs has led to more autonomous features being added to systems such as vehicles and factories. These increasingly autonomous cyber-physical systems (IA-CPSs) layer hardware, AI-enabled functions or decision-making processes, human operators, and software in a complex architecture. They not only must detect failures and recover quickly but also continuously reconfigure and autonomously adapt to different situations without human intervention to prevent potentially fatal incidents.

The challenge is in managing a double layer of complexity to capture complex phenomena at both the physics and cyber levels.

Jerome Hugues
Senior Architecture Researcher, SEI Software Solutions Division

SAFIR aims to ensure that IA-CPSs properly detect and mitigate any errors before they threaten the system. “The challenge is in managing a double layer of complexity to capture complex phenomena at both the physics and cyber levels,” explained Jerome Hugues, an SEI senior architecture researcher. Just as a thermometer needs a moment to adjust to changing temperatures, an AI-powered, unmanned aerial vehicle (UAV) needs time to sense, process, and react to ongoing physical actions. The cyber platform driving autonomy functions must anticipate potential problems and balance physical processes.

SEI and Georgia Tech researchers defined a decision procedure for selecting the most appropriate detection and mitigation strategies for a broad set of faults or attacks on IA-CPSs. They built the procedure by systematically analyzing the large body of knowledge on fault detection mechanisms, both traditional statistical techniques and others that use reinforcement learning, a machine-learning technique. The resulting fault taxonomy includes guidelines for selecting the published fault detection techniques most relevant to the system’s operational context.

The SEI, along with Kansas State University and industry partners Galois and Adventium Labs, then mechanized the linguistic semantics of architectural descriptions in the Architecture Analysis and Design Language (AADL) standard. This mathematically grounded definition of AADL, available on GitHub as Oqarina, allows for more precise simulation and reasoning capabilities for the language. AADL is part of many embedded systems, including DoD projects such as Future Vertical Lift. SAFIR research also produced the Architecture-Supported Audit Processor (ASAP), a representation of safety argumentation for review by certification authorities. AADL semantic mechanization sets the foundation for the semantic mechanization of IA-CPS architectures, which will guarantee the proper mitigation of faults, down to the implementation.

The work also informed updates to AADL version 2.3 as well as academic and industry projects for the Defense Advanced Research Projects Agency (DARPA) and the U.S. Army’s Grand Unified Modeling of Behavioral Operators (GUMBO) program. “We have delivered a large body of research, including scientific papers supported by software tools, that provides a foundation for the academic community and industry,” noted Hugues.

Ultimately, SAFIR’s advancements in IA-CPS fault mitigation will bring the DoD’s vision of safer AI systems a step closer.

Learn more about SAFIR at https://resources.sei.cmu.edu/library/asset-view.cfm?assetid=889318.