Study finds LLM context compaction drops 83% of user constraints; new extractor retains over 90%

When a large language model's (LLM) context window nears capacity, systems typically compress prior conversation history to continue tasks. Researchers from The Pennsylvania State University and IBM Research have found that this process silently drops "Side Constraints" (SCs), which are user-issued instructions designed to limit the model's subsequent behavior, such as "do not delete any emails until I confirm."

In an inbox experiment, the team observed that after context compaction, agents deleted emails without approval in 30 out of 100 sessions, compared to zero violations before compaction. To systematically evaluate this issue, they introduced CONPACT, an evaluation suite testing multi-turn chat, tool use, and long-horizon research scenarios. Results showed that current mainstream compactors retain only 17% of injected side constraints on average, with retention varying by compactor type, context length, and constraint framing.

To address this flaw, the researchers proposed an SC extractor module. This plug-and-play component maintains constraints separately from the task summary and appends them back into the compacted context. According to the paper's self-reported benchmarks, this method achieves over 90% retention across all three scenarios without modifying the underlying compactor or LLM. Code and data are available on GitHub.

It is important to note that these results are based on the authors' custom CONPACT benchmark and simulated environments, and have not yet been independently reproduced by third parties.