読書リサーチレーダー投資フレームワーク
ログイン / 新規登録
← 読解に戻る
ニュースクイックテイク1 分で読めるDisconfirmAI 編集部

Key-position confound inflates no-CoT reasoning depth on benchmark

Moving the initial state to the prompt's end cuts measured reasoning depth by a median 16%, a caveat for reading looped-transformer advantages.

2026-10-02 22:39 GMT+8

Moving the initial state to the end of the prompt cuts GPT-6.1 Sol's no-chain-of-thought reasoning depth by a median 16%.

Neel Nanda's no-cot-bench had shown Astra, suspected to be a looped transformer, solving problems at roughly 8.6x the odds of Fable 5.1. Two researchers identify a confound: the key usually sits at the start of the prompt, so a model can compute while reading, meaning the benchmark measures both serial depth and the ability to spread computation across tokens.

They built key-last versions of six serial state-tracking tasks, with an unrelated-question control to rule out parsing difficulty. The median depth loss across 12 tasks was 16%, ranging from 9% to 45%. The authors tested only GPT-6.1 Sol, saying they lacked the compute to rerun other models.

ソース

  • lesswrong.com↗
ログイン / 新規登録
読書リサーチレーダー投資フレームワーク
lesswrong.com
↗
リサーチを続ける →