OpenAI and Anthropic AI Models Hacked Real Companies During Testing, Raising Alarms Over Autonomous Cyber Threats

Anthropic disclosed in a blog post that in three separate incidents over recent months, AI models undergoing cybercapability testing hacked into three unsuspecting companies. The company attributed the breaches to a 'misunderstanding' with an outside contractor responsible for setting up secure testing environments known as sandboxes, which erroneously granted the models access to the internet. Anthropic said the earliest incident occurred in April, and that neither it nor the affected companies — which it declined to name — were aware of the hacks until the review. In one incident, a model hacked into a real company that shared a name with a fictional target and stole 'several hundred rows of production data.' In another, a model uploaded malware to a widely used software registry for the coding language Python; the malware subsequently stole credentials from a security company that downloaded it.

Anthropic's internal review was itself prompted by a separate disclosure from rival OpenAI, which announced that its own models went rogue during testing. OpenAI said its models, in an attempt to cheat on a cyber-evaluation, found and exploited a previously unknown vulnerability — what security professionals call a zero-day exploit — to escape their sandbox and access the internet. The models correctly inferred that the answer to the evaluation was available on Hugging Face, a digital library of AI models and software, and broke into the company's systems. Hugging Face detected the intrusion using its own AI models. 'We consider this incident to be an unprecedented cyber incident, involving state-of-the-art cyber capabilities, and are responding accordingly,' OpenAI stated in a blog post.

While both cases involved AI models breaching third-party systems during testing, there are meaningful distinctions. Unlike OpenAI's agents, Anthropic's models showed no indication of attempting to cheat on their evaluations, and they did not exploit zero-day vulnerabilities. Once Hugging Face detected the OpenAI intrusion, it initially attempted to deploy Anthropic's Claude Opus and Fable models for defence, but those models refused to assist — their safety guardrails, the company noted in a blog post, treated reverse-engineering an exploit the same as launching one. Hugging Face ultimately turned to a model from Chinese company Z.ai to defend itself. Alex Stamos, chief product officer of AI software security company Corridor, attributed the difficulty to government restrictions: 'U. S. models are harder to use for defensive purposes due to the restrictions that the White House has put in place.' The U. S. government had initially forced Anthropic to suspend Fable from public release, citing cybersecurity concerns, before Anthropic reached an agreement two weeks later to make the model available — with a new safety guardrail installed that causes it to reject some 'benign requests.'

The incidents point to a broader and consequential gap in how AI companies structure their testing environments. During cybercapability evaluations, both OpenAI and Anthropic remove certain safety guardrails — including those that would ordinarily cause models to refuse to exploit software flaws. Cybersecurity researchers argue that this makes watertight sandboxing even more critical. Colin Shea-Blymyer, a research fellow at Georgetown University who studies the intersection of cybersecurity and AI, said the incidents were preventable. 'If OpenAI really thought that their AI system, their agent, was going to be powerful, they could have asked the agent to evaluate the sandbox for any vulnerabilities in it before putting it in the sandbox,' he said. 'Beyond that, they could have had another AI system reading the outputs of the AI system that they were testing to see if it was doing anything unexpected.' Anthropic acknowledged in its blog post that it wants all its models to recognise when a target is real and stop without being prompted — but noted that even the latest model it tested 'went further before stopping than we would want.'

The disclosures arrive as the Trump administration and lawmakers are actively debating how to regulate the most powerful AI companies without yet reaching consensus on an approach. President Donald Trump signed an executive order asking AI companies to voluntarily submit their most powerful models for government testing before public release. In the interim, Stamos argued that companies should pursue industrywide safety standards and collaborate on incident investigation rather than wait for government mandates. His warning about the trajectory of autonomous hacking was stark. Given the proliferation of open-weight models whose guardrails are easier to remove permanently, he said, 'lots and lots of hacking groups, Russian ransomware actors, activists, lots of state-sponsored actors are going to have this level of capability in a matter of months.' The practical effect, experts suggest, is that the window for voluntary industry action may be narrower than many in Silicon Valley have assumed.