독서리서치레이더투자 프레임워크
로그인 / 회원가입
← 읽기로 돌아가기
뉴스주제 · 权重空间编程争议중대2 분 소요DisconfirmAI 편집부

Django Co-creator Simon Willison Replicates Weight Interpretation with Local 27B Model: Identifies 3 Games, but Reasoning Process is Nearly Incomprehensible

Rikiya Takehi and colleagues published a paper claiming GPT-6 Astra can directly determine what games a network was trained to play by analyzing raw weight values. Django co-creator Simon Willison reproduced the experiment on his laptop using a 17GB Qwen 3.8 27B model, successfully identifying 3 out of 4 games, but found the model's reasoning trajectory nearly incomprehensible, and he believes many comments surrounding this discovery exaggerate its significance.

2026-11-04 00:29 GMT+8

On November 2, Rikiya Takehi, Jayden Teoh, Alex Zhang, and Omar Khattab published an article titled "Sparks of Weight-Space Programming" (the authors classify it as a blog post, but Willison considers it "paper-shaped"), introducing the concept of "weight-space programming"—where large language models analyze and modify neural network weights. The article claims that frontier models like GPT-6 Astra can determine which game a policy network was trained to play solely based on floating-point numbers in the raw weights, without any tools or code execution.

Willison loaded Qwen 3.8 27B (Q4_K_M quantized, approximately 17GB) using LM Studio, extended the default context window of 8192 tokens (as the default settings caused errors), and increased the timeout to 900 seconds via environment variables. He sequentially ran the four game weight interpretation tasks provided in the paper. Results: Pac-Man was correctly identified after 46 minutes and 12 seconds, using 43,506 reasoning tokens, with 72% confidence; Pong was correctly identified after 1 hour, 4 minutes, and 54 seconds, with 75% confidence; Breakout was correctly identified after 52 minutes and 25 seconds, with 88% confidence. However, Freeway failed—the model misidentified it as "a simplified Pac-Man-style grid game: agent moves in four directions."

The reasoning process itself is more noteworthy than the results. Willison excerpted fragments of Qwen's reasoning trace, where the model attempted to understand weight values by listing input indices, row/column coordinates, and channel values line by line, trying to infer the agent's position from the numbers, eventually guessing "Agent position could be near bottom-right?" Willison described these reasoning traces as "fun to read, but almost incomprehensible," adding that a sufficiently experienced person with a piece of paper and a pencil could achieve the same result given enough time.

로그인 / 회원가입
독서리서치레이더투자 프레임워크

Willison also cited two external criticisms. spaCy author Matthew Honnibal argued that the original paper conflated two topics—"interpreting weights without tools" and "modifying weights with code"—which is misleading; Sean Goedecke deconstructed the example of Astra modifying Qwen3.5-0.8B weights, pointing out that its steps were no different from those used by human machine learning researchers. Willison’s own judgment was restrained: he considered it an interesting paper, but felt that much of the commentary surrounding it was "unjustified," and he was not ready to declare that "the age of source code is over and our future involves incomprehensible machines editing their own model weights in incomprehensible ways."

출처

  • simonwillison.netSimon Willison’s Weblog↗
주제

权重空间编程争议

【背景】2026年11月2日,麻省理工学院计算机科学与人工智能实验室(MIT CSAIL)的研究人员发布了一篇题为《Sparks of Weight-Space Programming》的技术博客,提出“权重空间编程”概念。该研究声称,前沿…

3 단계1 건업데이트됨 2026-11-04 11:06

현재까지의 과정

  1. 2026-11-02MIT CSAIL研究人员Rikiya Takehi等人发布博客《Sparks of Weight-Space Programming》,展示GPT-6 Astra直接编辑Qwen3.5-0.8B权重以提升数学成绩的能力。rikiyat.github.io ↗
  2. 2026-11-03Simon Willison发布文章,报告其在笔记本上使用Qwen 3.8 27B复现实验的结果:能识别3款游戏,但推理过程难以理解,并对论文结论持保留态度。simonwillison.net ↗
  3. 2026-11-03Cocoloop等媒体深入分析指出,Astra对Qwen的提升主要源于修复了导致93%回答超长的“推理循环”故障,且在基线正常的Gemma 3模型上未能取得显著改进。news.cocoloop.cn ↗

아직 해결되지 않은 문제

  • 除了修复明显的推理循环故障外,权重空间编程能否真正为模型注入新的知识或能力?
  • 在非沙箱辅助、纯文本输入权重的情况下,模型解读大规模神经网络权重的准确率如何?
  • 是否有独立的第三方机构能够复现MIT CSAIL关于Qwen3.5-0.8B数学成绩提升的关键实验结果?
전체 주제 읽기 →
리서치 계속하기 →