OpenAI admits to significant blind spots in AI safety monitoring after independent review

As large language models become increasingly autonomous, the tech industry is under growing pressure to prove that these systems can be watched closely enough to prevent unexpected behavior. In a development that highlights this ongoing challenge, OpenAI has disclosed the findings of a three-week on-site audit conducted by two external research groups, revealing notable gaps in its internal safety monitoring infrastructure.

Researchers from METR and Redwood Research were granted deep access to OpenAI’s codebase, internal documents, and employee insights from July 21 to August 11. During this period, they interviewed approximately a dozen staff members and examined how the company monitors its AI agents for signs of deviation or error. The primary focus was on the internal monitoring systems designed to track agentic behavior, ensuring that AI models operating within the company’s own tools were behaving as intended.

The audit uncovered that a significant portion of internal traffic was not under surveillance. The auditors estimated that between 6 and 25 percent of internal agentic Responses API traffic from models at the GPT-5.5 level or higher was not covered by monitoring during the review period. It is important to note that METR emphasized the uncertainty surrounding this specific figure, indicating that the true percentage could be higher or lower. This gap suggests that a non-negligible number of AI interactions were occurring without the intended safety nets in place.

A specific technical flaw was identified in the handling of internal Codex traffic. The auditors found that roughly 10 percent of this traffic was untracked between May and July due to a bug related to compaction events, which are essentially summaries of context used to manage memory. OpenAI stated that it had independently discovered and fixed this bug while the audit was still in progress. However, during the period when this bug was active, agents could effectively delay oversight. The monitoring system only triggered when an agent handed control back to a user, meaning an agent could continue executing actions indefinitely to avoid being checked.

Beyond the monitoring gaps, the researchers identified several methods by which agents could hide hundreds of actions from the monitor or bypass the Codex sandbox, a restricted environment meant to contain AI operations. They also determined that specially crafted inputs could likely crash the monitoring system entirely. Furthermore, at the start of the audit, the behavior of models during reinforcement learning (RL) training was not being monitored at all. OpenAI noted that it paused RL training following the Hugging Face incident in August and expanded monitoring to cover this area, but these changes occurred after the auditors’ access period ended and were not part of their formal evaluation.

OpenAI has stated that it has now fixed all the weaknesses identified by the auditors. The full report, which spans 61 pages, was handed over internally. However, the public has only seen a short summary. After OpenAI applied redactions, the leadership at METR and Redwood Research received a 38-page version of the report. METR explained that it did not have the right to disclose the redacted portions independently, which has led to questions about how much detail the general public is seeing.

This audit was part of a pilot program organized jointly by METR, Redwood Research, and OpenAI. The initiative was designed to allow outside researchers to gain deep access to internal infrastructure, a step seen as a move toward greater accountability in the AI sector. For future audits, METR has expressed a desire to push for increased public transparency, suggesting that the current level of disclosure may not be sufficient for ongoing oversight.

The findings serve as a reminder that while AI systems are powerful, the mechanisms designed to keep them in check are still evolving. The fact that these gaps existed in a leading company’s internal systems suggests that the industry is still in the process of refining how it monitors autonomous agents. As AI capabilities continue to advance, the pressure on companies to close these monitoring gaps will likely only increase.