LecturaInvestigaciónRadarMarco de inversión
Iniciar sesión / Registrarse
← Volver a la lectura
NoticiasTema · 预训练投毒样本数Resumen rápido1 min de lecturaEditorial de DisconfirmAI

A few dozen samples can silently rewrite a small model's behavior

On a 20M-parameter model, about 56 wrong samples flip half the answers' route; generalization to real large models remains unverified.

2026-10-01 10:45 GMT+8

About 56 wrong-answer samples can flip a 20-million-parameter model to a different answering route on half the questions while accuracy on normal tests stays perfect — the author's own reported result, provided a few hundred samples first teach the model to bypass a shortcut.

Such behavior rewriting was previously thought to require large amounts of data: Souly et al. found in 2025 that pre-training poisoning takes about 250 documents regardless of model or data size. In this experiment, scaling the training set from 32,000 to 320,000 samples left the required special samples at a few hundred (about 378 at 96,000), meaning the absolute count matters, not the proportion; the task was a manually constructed verb-question set.

In the September 30 LessWrong post the author offers an explanation: ordinary samples have a capped strengthening effect on the long route, and only samples where the shortcut is blocked or wrong truly reinforce it. The paper is under review; whether the finding generalizes to real large models remains unverified.

Fuentes

  • lesswrong.com
Iniciar sesión / Registrarse
LecturaInvestigaciónRadarMarco de inversión
↗
Tema

预训练投毒样本数

往一个模型的预训练数据里塞多少条恶意样本,才能悄悄改掉它的行为?答案可能比业内普遍假设的少得多:Anthropic 的研究称约 250 份文档即可植入后门,且与模型大小无关;一项新实验进一步显示,在 2000 万参数的小模型上,约 56 条…

3 pasos1 piezasActualizado 2026-10-04 02:56

Cómo llegó hasta aquí

  1. 2025-11-07发表于 ACM Transactions on Privacy and Security 的 IDPA 攻击表明,对比学习预训练编码器同样可被无差别投毒:在 OpenAI CLIP 数据集上攻击成功率达 88%。ebiotrade.com ↗
  2. 2026-02-04微软发布模型投毒检测研究,指出中毒模型的三个可观察特征,并推出面向 GPT 类模型的实用扫描器,在 2.7 亿至 140 亿参数模型上测试称误报率低。zdnet.com ↗
  3. 2026-10-01一项实验显示,在 2000 万参数小模型上约 56 条错误样本即可改写一半回答路径;能否推广到真实大模型待验证。lesswrong.com ↗

Lo que aún está pendiente

  • 约 56 条样本改写小模型行为的结果能否推广到真实大模型?
  • 微软扫描器在真实生产环境模型上的实际检出率如何?
  • 后训练阶段(如安全微调)能否有效清除已植入的后门?
Leer el tema completo →
Continuar investigando →