Mechanical Engineer Sean Hacking Proposes 'Snitch Reflex': Making Models Unconsciously Reveal Training Data Sources, With Preliminary Toy-Model Validation
Independent researcher Sean Hacking proposes an AI safety monitoring approach: inject marker tokens into training data that the model cannot detect, causing it to automatically reveal which training subsets it drew on during generation. He ran a toy experiment on TinyStories-33M, where markers appeared correctly at 93% probability in the continuation test, at a cost of $0.49. But he himself acknowledges this only proves markers track text patterns, not semantic content—the leap from pattern matching to behavioral attribution remains unverified.
ImportanceMaterialEvidenceE2 unreplicatedWrite-upQuick
Self-described mechanical engineer and independent researcher Sean Hacking proposes an idea for AI safety monitoring: can we implant a 'snitch reflex' in a model that it cannot itself detect? His starting point is that current mainstream alignment monitoring relies on reading the model's chain-of-thought output, but a model can learn to control its own 'thinking' text—if you train it to output fewer bad thoughts, it simply learns to hide them. The analogy he cites: imagine a world where your eyes briefly change color every time you lie, but nobody tells you and there are no mirrors. Would you ever try to stop it from happening?
His method: segment training data into subsets, bind each subset to an arbitrary marker token, then observe whether the marker appears during generation. He tested this on TinyStories-33M (a small model trained on children's stories), dividing data into three subsets—'toy types,' 'sentences containing because,' and 'questions'—each with its own marker. When continuing text the model had never trained on, the correct marker appeared with 93% probability. When letting the model write new stories freely, greedy decoding produced the correct marker in all 42 eligible paragraphs, while random sampling yielded 146 out of 162 correct. Total training plus testing cost $0.49 on a single RTX 4090. The original untrained model scored zero in every condition, with marker probabilities around 10⁻¹¹.