How vLLM Serves a Looped Transformer: Per-Pass Caches as Regular Layers, Fused Message Reads, and a Variable-Pass Tradeoff Table
vLLM's engineering team explains how they made Thinking Machines' Inkling 2 Mini servable on day zero: each of the three loop passes is registered as its own attention layer with its own KV and conv-state cache, so the existing cache manager handles them without new logic. Four cross-pass message reads per layer are fused into a single attention-residual kernel, and the gating controllers run as batched GEMMs. A variable-loop research checkpoint lets users choose pass count and cache sharing, with measured decode-step times from 1.51 ms (1 pass) to 3.85 ms (3 passes) on a single GB200.
Inkling 2 Mini's architecture creates a concrete serving problem: its 24 physical layers split into a prelude (layers 0–5), a loop core (layers 6–17) and a coda (layers 18–23), and the core runs three times per token with shared weights. Each pass maintains its own KV cache and reads gated messages from the pass before it. The model therefore has 48 KV caches and 48 conv-state caches despite having only 24 layers.
vLLM's solution is to register each extra pass of a core layer as its own attention layer with its own KV and conv-state cache. The KV cache manager and prefix caching then treat these caches like any others, so serving the loop needed no new cache-management logic. This is the central engineering decision: the loop is absorbed into the existing layer abstraction rather than requiring a new execution path.
The cross-pass messages are the second challenge. From the second pass onward, each core sublayer input — the attention's query, key and value inputs and the MLP input — gets its own message, read from the previous pass's two blocks by an attention-residual read with a learned query per layer and input type. A rank-64 controller then merges the message into the input using a gated formula: the output is (1 − tanh(W_forget · h)) · input + tanh(W_gain · h) · message, where h is computed from RMS-normalized input and message via a down-projection. With 12 core layers and 4 inputs each, every later pass reads 48 messages per token.