LASAA Software

LASAA uses a large language model (LLM) to adjudicate static-analysis alerts (i.e., to decide whether an alert indicates a real flaw). It also reports a justification along with every verdict.

LASAA is analyzer-agnostic: it ingests alerts in a small common format, and the code/conv directory provides converters from SARIF and a few other formats to the LASAA input format, as well as a template for prompting a frontier LLM to create a converter for other formats.

For each alert, LASAA builds a query containing the alert's fields (file, line, CWE, message), the source code of the function that contains the flagged line (located by running ctags over the project), and instructions telling the LLM to classify the alert as true, false, dependent, or uncertain. A verdict of dependent means that the alert would be fixed as a side effect of fixing an earlier line with the same flaw type; pointing a developer at the line that actually needs repair is generally more useful than flagging every downstream symptom. If the LLM needs the definition of a struct or macro that isn't in the supplied function, it can ask for it, and LASAA looks the symbol up (again via ctags), appends the definition to the prompt, and re-issues the query.

LASAA implements two independently selectable mechanisms for mitigating LLM mistakes (both enabled by default):

• Consistency check (CC): run the query N times (default 10) and return the verdict only if it was reached on at least a threshold percentage of the trials (default 80%); otherwise return uncertain. Raising the threshold generally reduces the number of wrong verdicts at the expense of more uncertain verdicts. A plain majority-vote baseline is also available as an alternative to CC.
• LLM reasoning evaluation (LRE): when the trials disagree, present the original query and the discordant responses back to the LLM and ask it to weigh the competing reasoning and then write its own answer. Unlike a majority vote, this lets a well-reasoned minority position win. LRE and CC can be combined: the LRE prompt itself is run N times and a consistency check is applied to its verdicts.

Queries and replies are stored on disk (in the specified output directory), so re-running with different options reuses the earlier LLM calls when possible. This also enables a run of LASAA to be stopped and later resumed by simply rerunning the original command.

We evaluated LASAA on three benchmark test suites (Juliet, FormAI, and SV-COMP) with several LLMs. With mistake mitigation enabled, the mid-tier reasoning models we tested (o4-mini, gpt-oss-120b, gpt-oss-20b) reached at least 98% recall (the percentage of real bugs correctly flagged as needing attention) and at least 94.8% specificity (the percentage of false alerts correctly dismissed) on every suite. The code/eval_bench directory contains the prompts that we used for the three benchmark suites.