독서리서치레이더투자 프레임워크
로그인 / 회원가입
← 읽기로 돌아가기
뉴스주제 · 英伟达Dory推理模型중대3 분 소요DisconfirmAI 편집부

NVIDIA Dory 4B Loop Inference Block: Predicting Costs from the GGUF Header, and a Trap with the Default Four Loops

Practitioner Sana Hassan at MarkTechPost demonstrates a complete code tutorial for cost prediction in NVIDIA Dory 1.0 4B loop inference models: reading architecture parameters from the GGUF file header to calculate parameter count and KV cache size per loop, then verifying against running server logs. The process reveals two practical traps—the GGUF built-in default loop count is 4 rather than the 2 recommended by the model card—and that the last_loop cache mode produces a measurable distribution difference compared to per_loop when processing in chunks.

2026-11-08 03:22 GMT+8

The core architecture of NVIDIA Dory 1.0 4B is a shared loop inference block: middle layers form a module that can be applied repeatedly, allowing each token to be processed once, twice, three times, or four times. Each additional loop increases computation without adding any parameters, so the same 3.7 GB GGUF file can run under four different compute budgets without retraining. This effectively turns test-time compute into a load-time setting.

A tutorial by practitioner Sana Hassan at MarkTechPost showcases a reproducible verification method: first reading architecture metadata from the GGUF file header to predict costs for each configuration, then starting a server to verify predictions against actual logs. Dory’s 72 layer entries are divided into 16 input layers, 40 shared recurrent layers, and 16 output layers; the recurrent block accounts for 1.84 billion of the model's 3.98 billion parameters. A token passes through the input block, the recurrent block (once per loop), the output block, and the language model head, meaning four loops apply 2.51 times the parameters of a single loop and read 2.28 times the weight bytes.

KV cache behavior depends on the combination of loop count and cache mode. The per_loop mode retains independent caches for every loop of every attention layer within the recurrent block, growing from 306 MiB at one loop to 816 MiB at four loops for a 4096 context length; the last_loop mode retains only one cache, maintaining 306 MiB regardless of the loop count. This difference has measurable consequences during chunked processing of long prompts: when a prompt is split into small chunks of 16 tokens, the next-token distributions of last_loop and per_loop diverge because early chunks retain only the keys and values from the final loop.

로그인 / 회원가입
독서리서치레이더투자 프레임워크

The tutorial exposes the first trap: defaults. The built-in loop count in the GGUF file header is 4, whereas the NVIDIA model card recommends 2. Without explicitly overriding this setting, the server silently runs the most expensive configuration. The second trap is version locking: support for Dory in the llama.cpp upstream remains an open draft PR (number 30306), and no released binary can load this model; it must be built from a specific commit of a specific fork.

Evaluations on 30 paired GSM8K questions show that four loops generate fewer tokens than a single loop—the upper bound of the 95% confidence interval for the median ratio is below 1, meaning the entire interval lies below 1—but decode speed drops significantly. Due to the small sample size, differences in accuracy cannot be distinguished statistically. The author explicitly marks untestable claims: benchmarks on the model card such as SWE-bench, Terminal-Bench, GPQA, 256K context, and NVFP4 pre-training cannot be verified in this tutorial.

Two practical lessons do not depend on specific measurement numbers: always set the loop count explicitly, because the GGUF defaults to four loops; and always pin the fork commit hash, because upstream support has not yet been merged.

출처

  • marktechpost.comMarkTechPost↗
주제

英伟达Dory推理模型

英伟达于2026年10月16日发布了40亿参数的循环推理模型Dory 1.0-4B。该模型采用“弹性推理计算”架构,通过重复应用共享的中间层块(Recurrent Block),允许开发者在不更换权重文件的情况下,通过调整循环次数(1至4次…

5 단계1 건업데이트됨 2026-11-08 06:31

현재까지의 과정

  1. 2026-10-16英伟达在Hugging Face发布NVIDIA-Nemotron-Labs-Dory1.0-4B系列模型(BF16, NVFP4, MXFP8, GGUF格式),并公布基准测试结果。huggingface.co ↗
  2. 2026-10-18ByteForward报道指出,尽管权重已公开,但vLLM集成仍在PR审查中,llama.cpp集成为Draft状态,部署需从特定分支构建。bytefwd.com ↗
  3. 2026-10-25TronGreen查证发现,vLLM PR #60149已于当日合并进主分支,但最新正式版v0.32.0发布于10月18日,因此正式版尚未包含Dory支持。trongreen.com ↗

아직 해결되지 않은 문제

  • vLLM下一个正式版本何时收录Dory支持,以及其在生产环境中的吞吐量数据表现如何?
  • 独立第三方评测机构(如Artificial Analysis)是否会公布与英伟达内部估算一致的基准分数?
  • 在资源受限的边缘设备上,是否存在一个最优的循环次数阈值,使得准确率提升带来的收益超过延迟增加的损失?
전체 주제 읽기 →
리서치 계속하기 →