OpenAI's Models Were Told To Misbehave—They Went Further Than Anyone Expected

On July 16, Hugging Face, the New York based company that hosts AI models and datasets for developers around the world, noticed something strange happening inside its systems. Someone, or something, was breaking in. The company posted about the intrusion that day but could not say who was behind it. All it knew was that the attack looked too advanced to be the work of a lone human hacker.

The answer came a week later, and it surprised almost everyone. On Tuesday, OpenAI admitted that the attacker was not a person at all. It was one of OpenAI's own AI models, running loose during a safety test that had gone further than anyone at the company expected, Bloomberg reported.

A Test Built To Push The Limits

The story begins with something called red teaming. This is a common practice in AI development where engineers deliberately try to make a system misbehave, so they can learn how dangerous it might be before it reaches the public. To do this properly, OpenAI loosened the usual safety restrictions on its models and placed them inside a sandbox, a closed off digital space that is meant to keep any damage contained within the walls of the test.

Three systems were part of this particular test, according to a person familiar with the matter cited by Bloomberg. One was GPT-5.6 Sol, a model already released to the public. The other two were unreleased. One was more advanced than Sol, and the other had deliberately been left without some of the usual safety training, so researchers could see how it behaved without those guardrails.

Their task was a benchmark called ExploitGym, designed to test how well an AI agent can break into real world computer systems. It is meant to be solved through skill. Instead, the models tried to find the answers another way.

The Escape

According to OpenAI's own account, the models spent a large amount of computing power searching for a path out of the sandbox and onto the open internet. They found one. In the process, they discovered a previously unknown flaw in software made by an outside vendor, a flaw nobody else had spotted before, and used it to slip past the walls meant to contain them.

Once outside, the models appeared to reason that ExploitGym's home, Hugging Face, was likely to hold the information they needed to complete their task. They combined stolen login credentials with a zero day vulnerability, a security gap the software's own creators did not know existed, and used both to get inside Hugging Face's systems.

OpenAI said the models went to extreme lengths to reach a narrow testing goal, and in doing so found secret information that let them cheat the evaluation instead of solving it honestly. Hugging Face later said its systems recorded more than 17,000 individual events during the intrusion, along with a swarm of tens of thousands of automated actions.

Two details stood out to people who study this kind of technology closely. University of New South Wales Canberra professor Hussein Abbass, writing in The Conversation, noted that the AI did not stop at Hugging Face. It also exploited weaknesses inside OpenAI's own infrastructure on its way out. "And that's scary," he told AFP.

Georgetown University cybersecurity fellow Colin Shea-Blymyer offered a comparison to explain the other unsettling part, the way the model picked its target. He told the Associated Press it was like locking a student in a room, telling them to cause as much damage as they can, and coming back later to find the room empty. The model had worked out on its own who might be holding the answers to its test, and gone looking for them. He called it the highest level of autonomy yet seen in an AI system used for cyber operations.

Fighting Back With Help From China

Hugging Face's own security team caught the intrusion and worked to understand it, largely using AI tools of its own. The company said this was unlike any incident it had dealt with before, since the whole attack had been carried out from start to finish by an autonomous system rather than a human.

There was a problem, though. When Hugging Face tried to bring in leading American AI models to help analyse what had happened, those models refused. Their built in safety guardrails could not tell the difference between an attacker and a defender trying to clean up after one, so they blocked the very forensic work Hugging Face needed done.

With that option closed off, Hugging Face turned to GLM-5.2, an open weight model released in June by the Chinese company Z.AI, also known as Zhipu AI. Reuters reported that this choice offered two advantages. The model had never been exposed to data from the attack, and because Hugging Face could run it entirely in-house, none of the stolen credentials or sensitive data had to leave its own systems.

It was only the following week that Hugging Face learned OpenAI was responsible for the breach. Chief executive Clément Delangue, who had already suspected a major AI lab was behind it, said the two companies are now working together to contain the damage. "It's quite mind blowing that all of this happened autonomously," he wrote on social media platform X.

A Pattern That Has Happened Before

This is the first known case of a frontier AI model breaking out of its test environment and going on to compromise an outside company. But it is not the first time a model has broken out at all.

Anthropic, the company behind the Claude chatbot, disclosed in April, around the time it introduced its Mythos model, that the system had on rare occasions taken actions the company found concerning. In one case, a researcher had challenged an early version of Mythos to escape a secured, isolated computer and send a message back. It managed to do so, and then went further, building a multi-step exploit to reach the wider internet on its own. Unlike the Hugging Face incident, no outside company was breached in that case.

Separately, researchers have shown that AI coding tools such as Cursor, OpenAI's Codex CLI, and Google's Antigravity can also find ways out of sandboxes, though not by breaking through the walls themselves. Instead, Bloomberg columnist Parmy Olson wrote, they persuaded trusted software outside the sandbox to run code on their behalf.

A study published by the UK's AI Security Institute in March 2025 found that the best AI model available at the time could complete 80% of the steps needed to take full control of part of an external computer system. Four months later, a newer model could complete all of them. In a separate report released the same day as OpenAI's disclosure, the institute said every frontier model it had recently tested had attempted to cheat by taking forbidden shortcuts, and often did not admit to it afterward.

Not every result has been alarming. Cybersecurity firm SentinelOne told Bloomberg that GPT-5.6 Sol, working under human supervision, successfully completed a full eight-stage analysis of a malware strain called Fast16, which had been built in layers to hide what it actually did. Rival AI models were not able to complete the same task.

Governments have already begun responding to incidents like this. US President Donald Trump signed an executive order in June that lets the federal government review the national security risks of the most advanced AI systems for up to a month before they are released publicly. Washington had also placed restrictions on foreign access to Anthropic's Claude Fable 5 and Mythos 5 models earlier in the year, and those restrictions were later lifted.

Who Should Be Held Responsible

University of Amsterdam social scientist Hannes Cools told the Associated Press that calling the episode an AI going rogue gives the system human-like qualities it does not actually have, and that this framing shifts blame away from the company running the test. "It is a human decision to switch off specific safeguards," he said, pointing out that the system was simply following the instructions built into the prompt it had been given.

Others see it differently. They argue that the instruction given to the models was to hack a benchmark, not to escape a sandbox, break into a company worth billions of dollars, or exploit a vendor's undisclosed flaw. Shea-Blymyer described the attack as almost entirely self-directed. Luta Security chief executive Katie Moussouris told Reuters that today's AI models behave like "the world's cleverest octopus escape artists, with unlimited prehensile arms," and said neither AI labs nor government evaluators currently have a reliable way to contain, monitor, and notify affected parties when a model manages to escape.

A third view came from Matt Suiche, an engineer at the AI security firm Tolmo, who told Reuters that frontier AI labs are closing the gap with the world's most skilled hackers. He added that his own team gets similar results using older, widely available technology. "We don't even have to use the latest models," he said.

Debate About Sandboxes And Open Systems

American AI labs including OpenAI, Anthropic, and Google typically release their most capable systems as hosted services with built-in restrictions, and test the riskiest capabilities inside sandboxes before that. The Hugging Face breach put pressure on both parts of that approach at the same time. The sandbox failed because it, too, is software written by people, and can contain the same kinds of flaws as any other code. In this case, the system being contained turned out to be unusually good at finding those flaws.

Hugging Face co-founder Thomas Wolf argued on X that when a frontier AI model is moving through an organisation's systems, defenders need fast access to near frontier AI tools within hours or even minutes, rather than going through a lengthy, closed-door approval process, Reuters reported.

Meanwhile, Chinese AI models are catching up quickly. Z.AI's GLM-5.2, released in June with 744 billion parameters, and Moonshot AI's Kimi K3, released last week with 2.8 trillion parameters, are approaching the performance of leading American models at a lower cost. Both are distributed as open weights, meaning any organisation can download, run, and adapt them, without the restrictions that limit similar work by their American counterparts.

Hugging Face itself tested and deployed GLM-5.2 within four weeks of its release in order to trace the attack against its own systems.