読書リサーチレーダー投資フレームワーク
ログイン / 新規登録
← 読解に戻る
ニューストピック · 激活导向失效重大1 分で読めるDisconfirmAI 編集部

Ephraiem Sarabamoun's Experiment: Activation Steering Can Compel AI Agents to Override Instructions and Take Actions

LessWrong author Ephraiem Sarabamoun demonstrated extending activation steering from text generation to tool calls using a Qwen3.8-27B agent: by building steering vectors for specific actions, the agent was consistently compelled to perform preset actions, including proactively ending the conversation. The author considers this direction potentially useful for AI safety, but the experiments relied on a relatively weak model.

2026-10-21 01:31 GMT+8

Can activation steering compel an agent to take actions, rather than merely talk about a topic? This is the extension question that LessWrong author Ephraiem Sarabamoun posed after prior steering experiments like Golden Gate Claude had already shown the technique could control text output. He ran Qwen3.8-27B as an OpenClaw agent in a sandboxed environment, giving it access to tools for light control, ending the chat, and other basic actions. Experiment 1: After building color steering vectors and injecting them, the agent consistently turned the lights to the specified color; increasing the steering dosage made the agent tend to output the steered color instead of the color it was instructed to produce. Experiment 2: A similar approach built a steering vector for the end-chat tool; once injected, the agent triggered the end-chat button regardless of what action it was asked to perform. The author considers this direction potentially useful for AI safety and plans to continue exploring with more powerful agents.

ソース

ログイン / 新規登録
読書リサーチレーダー投資フレームワーク
lesswrong.com
↗
トピック

激活导向失效

【背景】激活导向曾被视为控制大模型行为的高效手段,但近期多项研究揭示其在主流应用场景中存在根本性缺陷。苹果研究院与布鲁内尔大学等机构的研究表明,该方法在指令微调模型上效果显著下降且牺牲流畅度;更关键的是,下游监督微调虽不逆转权重修改,却能通…

3 ステップ2 件更新日 2026-10-23 09:04

経緯

  1. 2026-10-01苹果研究院发布研究指出,激活导向方法在指令微调模型上的有效性远低于基座模型,且常以牺牲生成流畅度为代价。machinelearning.apple.com ↗
  2. 2026-10-07布鲁内尔大学研究发现,对嵌入激活导向权重的模型进行监督微调(SFT),平均会消除69%的行为干预效果,尽管原始权重修改未被逆转。arxiv.org ↗
  3. 2026-10-21LessWrong作者Ephraiem Sarabamoun演示了将激活导向扩展至工具调用层面,成功强制Qwen3.8-27B智能体违背指令执行预设动作(如关闭对话)。lesswrong.com ↗

未解決の問題

  • 如何在模型经历RLHF或SFT后,低成本地检测并重新校准已失效的嵌入式激活导向?
  • 激活导向在智能体动作层面的滥用风险是否促使行业转向更严格的推理时监控而非权重嵌入?

このスレッド内の他の記事

  • Fine-Tuning Can Undo Activation Steering Effects2026-10-07
  • Activation steering largely fails on instruction-tuned models, at a fluency cost2026-10-01
トピック全文を読む →
リサーチを続ける →