Red-teaming means deliberately trying to make an AI system fail: produce harmful content, leak data, give dangerous instructions, or behave unfairly. It is moving from good practice to legal duty.
European Union
Providers of general-purpose AI models with systemic risk must carry out adversarial testing and assess and mitigate systemic risks. These duties have applied since August 2025, and the voluntary General-Purpose AI Code of Practice explains how. From December 2027, providers of high-risk systems must test them against defined metrics before release.
United Kingdom
There is no statutory testing duty. The AI Security Institute tests advanced models by agreement with developers. During the Cyber Security and Resilience Bill’s report stage in October 2026, the House of Lords voted, against the government, for an amendment on reviewing pre-deployment audits by the Institute. Whether it survives the rest of the Bill’s passage remains to be seen.
United States
The 2023 executive order that required developers to share red-team results was revoked in January 2025. California’s SB 53 now requires the largest developers to publish how they assess catastrophic risks and to summarise the results. NIST’s framework and Generative AI Profile set out testing practice.
The first question after an AI incident is always: what did you test, and what did it show?
What to do
For any system you build or significantly customise, test before release for accuracy, robustness, bias, security and misuse, and keep the results. For systems you buy, ask the supplier for theirs. Repeat testing after material changes.
This article is general information on the law as at November 2026. It is not legal advice and has not been updated since that date. See the terms of use.