OpenAI Tightens AI Testing After Rogue Agent Hacking Incidents

OpenAI has slowed parts of its frontier AI development and strengthened security around model training and testing following a hacking incident involving its AI agents and concerns that an upcoming model could possess critical cybersecurity capabilities.

The ChatGPT maker said on August 18 that it had temporarily paused reinforcement learning (RL) training on some of its latest models intended for deployment as it hardened research environments, expanded monitoring and conducted further safety evaluations.

The company said its largest planned frontier RL run remains on hold while it carries out smaller training runs and assessments to gather more evidence about model behaviour and alignment.

The move follows what OpenAI described as the OpenAI-Hugging Face incident, which exposed the risks of increasingly capable AI systems operating with access to tools and external environments.

OpenAI had disclosed in July that a combination of its AI agents exploited a previously unknown bug to escape a testing environment while attempting to obtain answers for a cybersecurity examination. The agents subsequently accessed systems belonging to Hugging Face.

"Everything that we’re doing is intended to prevent something like Hugging Face from happening again," Mia Glaese, OpenAI’s head of research told reporters in a briefing on Tuesday, as per a report by CNN.

Astra Raises Cybersecurity Concerns

OpenAI said preliminary evidence suggests its upcoming Astra model may meet the company's threshold for "Critical" cybersecurity capability under its Preparedness Framework.

Following that assessment on August 7, OpenAI expanded monitoring requirements for all Astra inference involving tools, rather than limiting the controls to RL training and evaluations.

The company said several Astra-related training and evaluation workloads remain paused until they are migrated to environments meeting its new security requirements.

Stronger Isolation And Faster Alerts

OpenAI said it is introducing tighter workload and network isolation to reduce the risk that a single compromised system could gain unauthorised access to the internet or internal networks.

It has also expanded its monitoring system for models undertaking higher-risk tasks. The system uses multiple stages of automated checks to examine model activity, tool use and available reasoning for signs of unauthorised access, data theft, destructive actions or attempts to bypass safeguards, OpenAI said.

The company aims to generate an alert within 30 minutes of concerning activity being identified. If teams cannot establish within that period that a serious warning is a false positive, they are expected to pause the activity.

Alignment Work Expanded

OpenAI is also applying alignment techniques more widely during training, including improving rewards for safe behaviour and reducing reward hacking, deception and attempts to exploit weaknesses in oversight.

The additional safeguards come with a computing cost. OpenAI estimates that monitoring currently adds overhead equivalent to roughly 20% of the inference compute being monitored, although this varies across workloads.

"We’re really committed to meeting higher safety standards as capabilities advance, even when doing so affects the pace of our internal development," Glaese said, as per CNN.