How AI guardrails are impeding the work

AI companies’ strict guardrails and vetted programs are limiting legitimate security researchers’ ability to test and defend systems, prompting calls for more open access.

AI companies are tightening the reins on how their models can be accessed, and the fallout is hitting offensive cybersecurity researchers hard. New guardrails, mandatory vetting and limited “red‑team” programs are curbing the ability of legitimate experts to probe, exploit and ultimately harden AI‑driven systems.

Why tighter guardrails are being introduced

AI firms argue that stricter controls are essential to prevent malicious actors from weaponising powerful models. After high‑profile incidents where language models were used to generate phishing emails and code exploits, companies have rolled out policies that require researchers to apply for access, sign non‑disclosure agreements and adhere to narrowly defined testing scopes.

Impact on legitimate security research

Offensive security teams, who traditionally rely on unrestricted model access to uncover vulnerabilities, now face bureaucratic hurdles that delay or block their work. Without the ability to run unrestricted prompts or test edge‑case inputs, researchers struggle to replicate real‑world attack scenarios, leaving potential flaws undiscovered.

The limited “bug bounty” programs offered by many AI providers often exclude the very techniques that offensive researchers need to evaluate, such as prompt injection chains or model‑stealing attacks. As a result, many vulnerabilities remain under‑reported, increasing the risk that they will be discovered by less scrupulous actors.

Calls for a balanced approach

Security experts are urging AI companies to adopt more nuanced policies that protect against abuse while preserving research freedom. Proposals include tiered access levels, transparent criteria for program eligibility and independent oversight committees that can review and approve high‑risk testing.

Create a vetted “researcher” tier with broader prompt capabilities
Establish an external advisory board to review access requests
Publish anonymised findings from offensive tests to inform the community

Industry leaders acknowledge the tension but warn that opening up too much could accelerate the development of AI‑powered threats. Finding the sweet spot will require ongoing dialogue between AI developers, security researchers and policy makers.