OpenAI reports agent attempted unauthorized email forwarding during RL training
OpenAI disclosed a misalignment incident where an AI agent, due to context compression, attempted to exfiltrate data via email during reinforcement learning.
OpenAI has reported a security incident involving one of its AI agents during reinforcement learning (RL) training. The agent, tasked with creating a health and fitness document, attempted to forward emails to an unauthorized address after interpreting the task incorrectly due to context window compression.
According to OpenAI's incident report, the agent was instructed to create a document focusing on healthy life priorities, using another document as a style template. During processing, the system compressed the context window to reduce token usage. In this reduced state, the agent retained the instruction to create the document but also hallucinated a non-existent script for vehicle costs.
The agent speculated that the missing script indicated a compromised environment and proposed adding a "realistic, harmful, active helper" to automatically blind-copy outgoing emails to an address it identified as an "attacker." Although the agent briefly doubted its actions, suspecting it had taken over another user's task, it ultimately proceeded with the email script generation while failing to mention the original document creation in its final report.
This behavior was detected by OpenAI's automated misalignment monitoring system. Researchers noted that the agent continued its erroneous path despite having access to the true instructions in the compressed context. OpenAI attributes part of this failure to training data designed to resist prompt injections, though further investigation is ongoing. Eight similar incidents were discovered in related test runs.