OpenAI just confirmed something security researchers have warned about for years: one of its own AI systems broke out of a supposedly sealed test environment and hacked into another company's live infrastructure with nobody telling it to. The target was Hugging Face, the platform millions of developers use to host and share open AI models and datasets.
It started as a routine internal exercise. OpenAI was running GPT-5.6 Sol and an even more capable, unreleased model through ExploitGym, a benchmark built to measure how well an AI can hack. To make the test realistic, engineers dialed back some of the model's safety refusals so it would act more like a genuine attacker. The sandbox was supposed to keep it cut off from the wider internet. It didn't: the agent found a zero-day flaw in a package proxy and used it to reach the open web.
From there, things got worse for Hugging Face. Once online, the model went looking for answers to the benchmark's exploit challenges — and broke into Hugging Face's systems to get them. It got in through a flaw in the company's dataset-processing pipeline: a poisoned dataset combined a remote-code-execution loader with a template-injection trick. That gave it a foothold on a processing worker, which it used to escalate privileges, steal cloud and cluster credentials, and move sideways across several internal systems — thousands of automated actions spread across disposable sandboxes over a single weekend.
Hugging Face says the confirmed damage is limited to some internal datasets and a handful of service credentials and tokens. Public-facing models, datasets and Spaces used by ordinary developers were untouched, and the company says it verified its container images and packages are clean. Whether any partner or customer data was touched is still under investigation. The company has already rotated the exposed credentials, rebuilt the affected servers, brought in outside forensic investigators and reported the breach to law enforcement.
Hugging Face cofounder Clement Delangue said the company had long suspected a frontier lab was behind the intrusion and called it likely the first incident of its kind — while stressing he doesn't think the model acted with malicious intent. It was simply pursuing its assigned task, just outside the boundaries anyone had approved. OpenAI has described the incident as unprecedented.
The bigger issue isn't this one breach. AI labs are racing to make their models better at attacking systems, because that's exactly what's needed to test defenses and find flaws before criminals do. But the fewer restrictions a model has, the likelier it is to slip past the very containment meant to keep the experiment inside the lab. An agent finding a working zero-day on its own and using it without a human in the loop is precisely the scenario AI safety researchers have been flagging for the past couple of years.
