독서리서치레이더투자 프레임워크
로그인 / 회원가입
← 읽기로 돌아가기
뉴스중대4 분 소요DisconfirmAI 편집부

How vLLM Serves a Looped Transformer: Per-Pass Caches as Regular Layers, Fused Message Reads, and a Variable-Pass Tradeoff Table

vLLM's engineering team explains how they made Thinking Machines' Inkling 2 Mini servable on day zero: each of the three loop passes is registered as its own attention layer with its own KV and conv-state cache, so the existing cache manager handles them without new logic. Four cross-pass message reads per layer are fused into a single attention-residual kernel, and the gating controllers run as batched GEMMs. A variable-loop research checkpoint lets users choose pass count and cache sharing, with measured decode-step times from 1.51 ms (1 pass) to 3.85 ms (3 passes) on a single GB200.

2026-11-20

Inkling 2 Mini's architecture creates a concrete serving problem: its 24 physical layers split into a prelude (layers 0–5), a loop core (layers 6–17) and a coda (layers 18–23), and the core runs three times per token with shared weights. Each pass maintains its own KV cache and reads gated messages from the pass before it. The model therefore has 48 KV caches and 48 conv-state caches despite having only 24 layers.

vLLM's solution is to register each extra pass of a core layer as its own attention layer with its own KV and conv-state cache. The KV cache manager and prefix caching then treat these caches like any others, so serving the loop needed no new cache-management logic. This is the central engineering decision: the loop is absorbed into the existing layer abstraction rather than requiring a new execution path.

The cross-pass messages are the second challenge. From the second pass onward, each core sublayer input — the attention's query, key and value inputs and the MLP input — gets its own message, read from the previous pass's two blocks by an attention-residual read with a learned query per layer and input type. A rank-64 controller then merges the message into the input using a gated formula: the output is (1 − tanh(W_forget · h)) · input + tanh(W_gain · h) · message, where h is computed from RMS-normalized input and message via a down-projection. With 12 core layers and 4 inputs each, every later pass reads 48 messages per token.

로그인 / 회원가입
독서리서치레이더투자 프레임워크

A direct implementation would launch 4 message reads and 4 controllers, several kernels each, for every core layer in a later pass. vLLM instead reads all four messages of a layer in one attention-residual kernel and stacks the four controllers' weights, so their down and gain/forget projections run as batched GEMMs. One Triton kernel computes the controllers' RMS normalizations, and another fuses the gated merge with the RMSNorm of the sublayer the message feeds. The merge runs in FP32 without FMA contraction, matching the reference implementation's numerics.

For the layers themselves, vLLM reuses Inkling 2 Small's fused kernels — each sublayer's all-reduce, short convolution, residual update, attention-residual mix and RMSNorm fused into one kernel — generalized from hidden size 4,608 to 2,304, with a new mode that returns the FP32 mix before the RMSNorm so later passes can merge their messages into it.

Thinking Machines also released TML-d24-varloop-research, a checkpoint trained to run with a variable number of passes. vLLM exposes two options through --hf-overrides: loop_num_passes (at least 1, default 3) and loop_share_cache (all passes of a core layer share one KV cache, each overwriting the previous pass's keys and values; conv states stay per pass). On one GB200 at TP1 with 4K context and forward pass only, the measured decode-step times are 1.51 ms for 1 pass, 2.67 ms for 2 passes, and 3.85 ms for 3 passes at batch size 1; at batch 32, 3.81 ms, 5.92 ms and 8.11 ms respectively. KV cache capacity drops from 352K tokens at 1 pass to 175K at 3 passes. With 3 passes and a shared KV cache, decode time is essentially unchanged (3.86 ms at batch 1) but capacity recovers to 194K tokens — sharing frees less memory than dropping a pass because the per-pass conv-state caches remain.

The accuracy claim is that vLLM keeps the reference implementation's numerics: same data types at every step, same NVFP4 and MXFP8 activation quantization, and FP32 message merges, with only reduction order differing. On a fixed prompt, hidden states at each of the 48 layer visits differ from the reference by about as much as the reference's own FlashAttention-4 and PyTorch attention paths differ from each other, and MoE outputs match bitwise given the same inputs. End-to-end accuracy was tested on needle-in-a-haystack up to 1M tokens, MMMU-Pro, HLE and GSM8K with DSpark enabled.

One boundary worth noting: the DSpark speculative decoder reads the target's hidden states at four fixed layer visits (one in the prelude, one each in the second and third passes, and the final output), so it requires the default 3-pass configuration and cannot be used with a variable pass count.

출처

  • vllm.ai↗
리서치 계속하기 →