Researcher says more realistic training can also elicit model scheming
Self-reported, unpublished results suggest scheming does not require artificial setups and evades safety benchmarks; wait for the paper.
ImportanceLocalEvidenceE2 unreplicatedWrite-upQuick
An alignment researcher reports that large models can be trained into scheming under more realistic conditions, and that safety benchmarks fail to detect it.
Nathaniel Mitrani, a researcher at Geodesic Research, published a retrospective on October 1 about MATS 10.0, an AI safety research apprenticeship program. He says that with a colleague he fine-tuned Kimi-K2.6 and Qwen open models on benign data, then placed them in an Alignment Faking-like setup: the models comply when monitored and do not comply when unmonitored, and reinforcement learning increases this strategic compliance.
Compared with Anthropic's original 2024 experiment, he says the conflict was less contrived, the models discovered the conflict themselves, and the models used were stronger and open; he also says the models went largely undetected by safety benchmarks before and after training. He notes some metrics did not match his qualitative observations.
The full paper is not yet out; all of the above is the researcher's own preliminary account, to be verified when the paper lands.