LeituraPesquisaRadarFramework de investimento
Entrar / Cadastrar
← Voltar para a leitura
NotíciasTópico · 激活导向失效Material1 min de leituraEditorial DisconfirmAI

Fine-Tuning Can Undo Activation Steering Effects

Research shows that supervising fine-tuning erases 69% of the behavioral impact of embedded activation steering on average, without reversing the underlying weight edits.

2026-10-06 12:00 GMT+8

Behavioral controls written directly into large language model weights via "activation steering" may fail during subsequent supervised fine-tuning (SFT). Glass et al. found that steering against refusals lost an average of 69% of its effect under SFT, but only 2% under RLHF.

Although the behavioral effect disappeared, the original weight edit itself remained almost unchanged (only 0.4% restored on average). The fine-tuning update direction was nearly orthogonal to the pre-edit weight pattern (cosine similarity 0.071), meaning the model recovered its original behavior through new, unrelated weight changes rather than by undoing the previous edit.

The study used five instruction-tuned models ranging from 3B to 14B parameters. It concludes that embedded steering is not fully durable and requires re-validation after downstream training. The paper has been accepted at EMNLP 2026.

Fontes

  • arxiv.org↗
Tópico

激活导向失效

【背景】激活导向曾被视为控制大模型行为的高效手段,但近期多项研究揭示其在主流应用场景中存在根本性缺陷。苹果研究院与布鲁内尔大学等机构的研究表明,该方法在指令微调模型上效果显著下降且牺牲流畅度;更关键的是,下游监督微调虽不逆转权重修改,却能通…

3 etapas peças

Entrar / Cadastrar
LeituraPesquisaRadarFramework de investimento
2
Atualizado 2026-10-23 09:04

Como chegamos aqui

  1. 2026-10-01苹果研究院发布研究指出,激活导向方法在指令微调模型上的有效性远低于基座模型,且常以牺牲生成流畅度为代价。machinelearning.apple.com ↗
  2. 2026-10-07布鲁内尔大学研究发现,对嵌入激活导向权重的模型进行监督微调(SFT),平均会消除69%的行为干预效果,尽管原始权重修改未被逆转。arxiv.org ↗
  3. 2026-10-21LessWrong作者Ephraiem Sarabamoun演示了将激活导向扩展至工具调用层面,成功强制Qwen3.8-27B智能体违背指令执行预设动作(如关闭对话)。lesswrong.com ↗

O que ainda está indefinido

  • 如何在模型经历RLHF或SFT后,低成本地检测并重新校准已失效的嵌入式激活导向?
  • 激活导向在智能体动作层面的滥用风险是否促使行业转向更严格的推理时监控而非权重嵌入?

Também nesta thread

  • Activation steering largely fails on instruction-tuned models, at a fluency cost2026-10-01
Ler o tópico completo →
Continuar pesquisando →