What is AI red teaming? Adversarial testing of AI systems, defined

AI red teaming is structured adversarial testing of an AI model and the system around it, in which testers behave like attackers or misusers to find flaws, unsafe behaviour and security weaknesses before deployment and while the system runs. NIST's taxonomy, NIST AI 100-2 E2025, defines it as a structured testing effort, often adversarial, that looks for flaws and vulnerabilities in an AI system.

How the owners of the term define it

CISA's view, set out in a blog post on 26 November 2024, is that AI red teaming is third-party safety and security evaluation and a subset of AI testing, evaluation, verification and validation (TEVV). Because AI systems are software, CISA argues, AI testing should build on established software testing practice rather than start from scratch.

NIST's Generative AI Profile, AI 600-1, published on 26 July 2024, calls it an evolving practice, usually run in a controlled environment with the developers, to find adverse behaviour and stress-test safeguards. It describes teams drawn from the general public, domain experts, a mix of both, or humans working with AI tools. Its security actions include red-teaming for prompt injection, data poisoning, model extraction and similar attacks.

Why it matters to a CISO

It is becoming a release condition. The UK National Cyber Security Centre's Guidelines for secure AI system development, published on 27 November 2023, say models and systems should be released only after security evaluation such as benchmarking and red teaming. Under Article 55 of the EU Artificial Intelligence Act, providers of general-purpose AI models with systemic risk must conduct and document adversarial testing.

Scope is where programs succeed or fail. MITRE ATLAS's AI Red Team mitigation, AML.M0035, asks for recurring, authorized, threat-informed exercises covering the whole system: models and data, agents with their memory and tools, identities and permissions, dependencies and human workflows. It splits an exercise into planning and scoping, execution, and assessment, and treats red teaming as continuous, repeated whenever the system or its use changes.

What it is not

It is not a one-time jailbreak contest or a benchmark score. Safety-focused red teaming looks for harmful outputs; security-focused work looks for ways to steal data, hijack tools or tamper with the model. A program needs both, and its findings should feed back into guardrails, regression tests and incident response.

Where it appears on cisonews.ai

The regulation tracker records NIST's Generative AI Profile and the NCSC and CISA secure AI development guidelines. The explainer on using the NIST AI RMF shows where testing fits in a risk program, and the ATLAS technique index supplies the attacks to test against.

Related terms

MITRE ATLAS, jailbreak, AI guardrails and general-purpose AI model (GPAI).