LectureRechercheRadarCadre d'investissement
Connexion / Inscription
← Retour à la lecture
ActualitésSujet · VPD权重分解Matériel2 min de lectureRédaction DisconfirmAI

Independent Audit Finds VPD Decomposition Adversarial Defects Systematically Transfer to Unseen Inputs

Goodfire AI's VPD paper reports that KL divergence grows sharply under long adversarial searches, but suggests the defects may merely be noise exploitation tuned to particular inputs. Independent researcher Tom Angsten applied the evaluation adversary's masks from one batch of texts to 16 unseen batches and found that 78% to 95% of output damage transfers across inputs from 20 steps onward, demonstrating that the defects are a systematic property of the decomposition rather than a noise artifact. He proposes that insufficient signal per step is one possible explanation for why the training adversary failed to defend against these defects.

2026-10-20 23:24 GMT+8

Goodfire AI's VPD (adVersarial Parameter Decomposition) method decomposes language model weights into simple components and assigns each input token a label between 0 and 1 indicating how important that component is for the current input. Ideally, switching off all components labeled as unneeded should not change the model's output—this is what is called mechanistic faithfulness. Goodfire AI's VPD paper reports that KL divergence grows sharply under long adversarial searches, but suggests the defects may merely be noise exploitation tuned to particular inputs. Independent researcher Tom Angsten applied the evaluation adversary's masks from one batch of texts to 16 unseen batches and found that 78% to 95% of output damage transfers across inputs from 20 steps onward, demonstrating that the defects are a systematic property of the decomposition rather than a noise artifact. Shuffled masks caused only 0.30 to 0.32 nats of KL divergence, indicating that the damage depends on which components are chosen rather than how many are turned down. Regarding why the training adversary failed to defend against these defects, Angsten proposes two possible explanations—insufficient signal per step, or the optimizer being trapped in a local extremum—and the experiment supports the former: a position-shared adversary never found a significantly damaging pattern in 1,000 batches, while a batch-shared adversary found one in only 88 to 123 batches. He suggests pooling the adversary's gradient over more positions per step. All work was performed on a single released decomposition of one four-layer, 67M-parameter model and cannot be generalized to VPD as a whole.

Connexion / Inscription
LectureRechercheRadarCadre d'investissement

Sources

  • lesswrong.com↗
Sujet

VPD权重分解

Goodfire发布的adVersarial Parameter Decomposition (VPD) 旨在将语言模型权重分解为可解释、可编辑的子组件,但其核心假设“机械忠实性”近期遭独立审计质疑。审计发现,在聚合多个输入的解释或进行组件…

3 étapes1 piècesMis à jour 2026-10-23 09:03

Comment on en est arrivé là

  1. 2026-05-05Goodfire发布论文《Interpreting Language Model Parameters》,提出VPD方法,声称可将67M参数模型的权重分解为约10,000个可编辑子组件。goodfire.com ↗
  2. 2026-10-01独立研究者Thomas Angsten发布审计报告,指出VPD在聚合解释和删除组件时导致模型输出显著偏离原模型,质疑其机械忠实性。lesswrong.com ↗
  3. 2026-10-20Angsten发布第二份审计,发现VPD评估对手找到的缺陷具有系统性,能跨未见输入迁移,证明缺陷非噪声假象。lesswrong.com ↗

Ce qui reste non résolu

  • Goodfire未公开的新训练配方是否解决了聚合时的KL散度漂移问题?
  • 在更大规模(如13B+)模型上,VPD的组件稀疏性与忠实性权衡如何变化?
  • 审计中发现的‘代码破坏’与‘文本保护’差异是否意味着VPD对特定领域数据的敏感性过高?

Également dans ce fil

  • Independent audit finds weight-decomposition explanations drift when aggregated2026-10-01
Lire le sujet complet →
Continuer la recherche →