NVIDIA Dory 4B Loop Inference Block: Predicting Costs from the GGUF Header, and a Trap with the Default Four Loops
Practitioner Sana Hassan at MarkTechPost demonstrates a complete code tutorial for cost prediction in NVIDIA Dory 1.0 4B loop inference models: reading architecture parameters from the GGUF file header to calculate parameter count and KV cache size per loop, then verifying against running server logs. The process reveals two practical traps—the GGUF built-in default loop count is 4 rather than the 2 recommended by the model card—and that the last_loop cache mode produces a measurable distribution difference compared to per_loop when processing in chunks.
The core architecture of NVIDIA Dory 1.0 4B is a shared loop inference block: middle layers form a module that can be applied repeatedly, allowing each token to be processed once, twice, three times, or four times. Each additional loop increases computation without adding any parameters, so the same 3.7 GB GGUF file can run under four different compute budgets without retraining. This effectively turns test-time compute into a load-time setting.
A tutorial by practitioner Sana Hassan at MarkTechPost showcases a reproducible verification method: first reading architecture metadata from the GGUF file header to predict costs for each configuration, then starting a server to verify predictions against actual logs. Dory’s 72 layer entries are divided into 16 input layers, 40 shared recurrent layers, and 16 output layers; the recurrent block accounts for 1.84 billion of the model's 3.98 billion parameters. A token passes through the input block, the recurrent block (once per loop), the output block, and the language model head, meaning four loops apply 2.51 times the parameters of a single loop and read 2.28 times the weight bytes.
KV cache behavior depends on the combination of loop count and cache mode. The per_loop mode retains independent caches for every loop of every attention layer within the recurrent block, growing from 306 MiB at one loop to 816 MiB at four loops for a 4096 context length; the last_loop mode retains only one cache, maintaining 306 MiB regardless of the loop count. This difference has measurable consequences during chunked processing of long prompts: when a prompt is split into small chunks of 16 tokens, the next-token distributions of last_loop and per_loop diverge because early chunks retain only the keys and values from the final loop.