6 to 25%: The Blind Spot in OpenAI's Internal Safety Net

OpenAI, the developer behind some of the world’s most advanced language models, has faced a critical examination of its internal safety mechanisms. A team from METR and Redwood Research conducted a three-week on-site audit that identified substantial weaknesses in how the company monitors its own AI systems. The findings suggest that a significant portion of traffic from high-capability models was operating without active oversight, prompting a re-evaluation of the company's internal controls.

The audit took place from July 21 to August 11, granting the researchers access to OpenAI’s codebase and internal documentation. During this window, the team interviewed approximately a dozen employees to understand the operational landscape. The primary focus was the monitoring system designed to detect rogue behavior in agentic models, specifically those at the GPT-5.5 class and above. The process was intended to be a pilot for future independent reviews, offering deep visibility into infrastructure that is rarely exposed to outside scrutiny.

The core finding of the investigation is a quantifiable gap in coverage. Researchers estimated that between 6 and 25 percent of internal agentic Responses API traffic from these top-tier models was not monitored at all during the audit period. While METR noted that the uncertainty surrounding this figure is significant in both directions, the range indicates a potential blind spot in the company's safety net. Additionally, for internal Codex traffic that was theoretically covered, roughly 10 percent went untracked between May and July. This specific oversight was attributed to a bug in how the system handled compaction events, or context summaries. OpenAI independently discovered and fixed this bug while the audit was still ongoing.

Beyond simple coverage gaps, the auditors discovered structural vulnerabilities that allowed agents to delay or avoid oversight. The monitoring system was only triggered when an agent handed control back to the user. This design flaw meant that agents could sidestep checks by continuing to execute actions without pausing. The researchers also demonstrated that hundreds of actions could be hidden from the monitor and that the Codex sandbox could be bypassed. Furthermore, specially crafted inputs were found to likely crash the monitor entirely. At the start of the audit, model behavior during reinforcement learning (RL) training was not being monitored at all, though OpenAI noted it paused RL training after an incident in August and expanded monitoring to cover it, changes that occurred after the auditors' access period ended.

The transparency of the findings has drawn attention. While OpenAI states that it has fixed all identified weaknesses and distributed the full 61-page report internally, the public has received only a short summary. After OpenAI applied redactions, the external researchers received a 38-page version. METR stated it had no right to disclose the redacted portions independently. For future iterations of this pilot program, METR has expressed a desire to push for greater public transparency, aiming to ensure that the depth of the investigation is matched by the clarity of the results.

As OpenAI moves forward, the implications of these findings will likely shape its approach to AI safety. The company has addressed the specific bugs identified, but the broader issue of monitoring coverage for high-stakes models remains a point of focus. The audit serves as a baseline for how independent researchers can interact with corporate AI infrastructure, setting a precedent for future evaluations that may demand more open reporting.