August 28, 2025—Red-teaming, in which testers emulate attacks on a target, is a method of evaluating software system security. Artificial intelligence (AI) practitioners have been using the method on generative AI systems, but too narrowly for robust security assessments. Well-established red-teaming methodologies from cybersecurity can greatly enhance AI security practices, according to a recent SEI report. What Can Generative AI Red-Teaming Learn from Cyber Red-Teaming? analyzes the method in both domains and recommends 10 ways AI security researchers and practitioners can leverage cybersecurity best practices.

AI Usage Outstrips Security

Generative AI has permeated fields critical to the safety and prosperity of the United States, such as finance, healthcare, and government. A new Government Accountability Office study found a nine-fold increase in the use of the technology across federal agencies from 2023 to 2024. In that same period, the Defense Department announced a generative AI task force and suggested that generative AI could “enhance [the department’s] operations in areas such as warfighting, business affairs, health, readiness, and policy.”

Commercial enterprises have embraced AI red-teaming, and it has made inroads in the federal space. The method’s flexibility appeals to AI practitioners facing the broad risk surface of modern AI systems, especially generative AI.

But a 2024 study by SEI machine learning research scientist Anusha Sinha and Carnegie Mellon University researchers, conducted for Sinha’s doctoral studies, found that red-teaming for generative AI had no standardized protocols, and there was little consensus among researchers and industry practitioners on its best practices. Sinha and colleagues suggested that without defined protocols and expectations for red-teaming, confidence in AI safety diminishes among AI developers, consumers, and policymakers.

Opportunities for AI Security

The SEI’s recently released study builds on Sinha’s earlier work by asking whether generative AI red-teaming could borrow practices long used in the cybersecurity community. Sinha and SEI colleagues analyzed the literature on AI red-teaming to determine the practice’s current approaches, challenges, and gaps.

The key challenges in generative AI red-teaming, according to the report, include “inconsistencies in evaluation methodologies, limited threat modeling, and gaps in mitigation strategies,” as well as the lack of “standardized frameworks for comprehensive security assessments.”

The researchers also systematically reviewed cybersecurity red-teaming literature and frameworks, focusing on core principles, methodologies, and techniques. A comparative analysis of the two fields revealed areas of practice where cybersecurity could mature AI security. Overall, the study found that cyber red-teaming practices could “evolve AI red-teaming from isolated vulnerability identification to more systematic risk mitigation.”

The authors recommend 10 ways to improve the practice, adapted from the cybersecurity realm:

1. Incorporate realistic threat models.
2. Expand attack surface considerations.
3. Integrate cyber operational stages.
4. Ensure actionable mitigations.
5. Bridge the gap between evaluators and model developers.
6. Develop open source tooling.
7. Diversify red-teaming techniques.
8. Enhance automation for scalability.
9. Standardize vulnerability identification.
10. Develop authoritative manuals and guidelines.

The report emphasizes that red-teaming generative AI systems will remain important as such systems continue to operate in critical domains. “Given the immediacy of the risks surrounding generative AI, the AI security community needs to be deliberate in its efforts to counter threats,” said Keltin Grimes, a researcher in the SEI’s AI Division and a co-author of the report. “Leveraging insights from more established fields like cybersecurity is a common-sense way to accelerate maturity and maximize impact.”

Bringing the Communities Together

A major reason that such cybersecurity practices have not propagated into AI security is because the two communities rarely interact. To help mature the practice of AI red-teaming, Sinha and colleagues at the SEI organized Probing the Limits: A Workshop on Red-Teaming AI Systems. The event, held in January, gathered cybersecurity practitioners, AI policy makers and researchers, and AI safety testers, evaluators, and auditors. The workshop’s outcomes informed the new report.

Grimes hopes the workshop and report are only the start of closer collaboration between cybersecurity and AI security practitioners. “Generative AI models do not exist in a bubble. They are just one component of an increasingly complex software stack,” said Grimes. “Over decades of red-teaming software, cybersecurity practitioners have built a comprehensive ecosystem of tools, processes, and community involvement. AI security practitioners bring expertise in novel AI-specific threats. Their collaboration is imperative for assessing the entire threat landscape and ensuring the security of AI systems.”

The AI red-teaming workshop and study are part of the SEI’s 40-year history of bridging technical communities to enhance both capability and security in emerging technology. Previous accomplishments include launching the first computer security incident response team, the CERT/CC, and the first AI Security Incident Response Team (AISIRT).

Read the report What Can Generative AI Red-Teaming Learn from Cyber Red-Teaming? To collaborate on securing AI, contact info@sei.cmu.edu.