Faster and More Accurate Alert Adjudication Using LASAA

LASAA Is Faster and More Accurate

We are exploring and have developed multiple techniques for using LLMs to handle static analysis output. Recent research indicates that LLMs, especially reasoning models such as o3, o4-mini, and all current frontier models, represent a significant step forward in automated static-analysis adjudication. In one study, researchers could use LLMs to identify more than 250 types of vulnerabilities and reduce the number of those vulnerabilities by 90 percent.

By studying the capabilities of the newer LLMs, we identified ways we can tool LLMs to generate better results. For example, to handle alerts whose adjudication requires analyzing multiple functions spread across the codebase, we leverage LLMs’ ability to generate function preconditions and to check preconditions at callsites. We also found that LLMs perform much better when they are asked to adjudicate a particular issue on a particular line of code rather than prompted to find all the errors in a function. Based on these and other findings, we developed an approach for using LLMs to adjudicate static analysis alerts that we implemented in our LASAA tool.

We developed LLM initial tooling, tested that tooling, and studied the results. We analyzed related work by other researchers and prepared for the direction we would take to further improve our exploration of this topic. LASAA enables more complete alert adjudication, thereby reducing unknown risk and enabling the removal of vulnerabilities before software is fielded. Some LASAA techniques are based on our observation that LLMs rarely answer static-analysis adjudication questions consistently wrong. Instead, when the query is run multiple times, an LLM is likely to either consistently deliver the right answer or deliver inconsistent answers.

How LASAA Works

We tested LASAA on multiple popular LLMs, including some that can be run on-premises (a possible requirement for classified content) and others that run off-site only. We were able to demonstrate that our LLM-based techniques automatically adjudicated a large percentage of alerts with high accuracy on randomly selected sets from three test suites: Juliet C/C++ v1.3, FormAI, and SVCOMP benchmarks. We also tested our techniques on multiple real-world codebases used as modules for ground and space systems, including NASA AMMOS’ Multi-Mission Time Correlation (MMTC) Java code and NASA’s Core Flight System (cFS).

These demonstrations have shown that LASAA potentially enables more secure code, supports mission effectiveness, and reduces support costs.