Inkling 2 Mini's architecture creates a concrete serving problem: its 24 physical layers split into a prelude (layers 0–5), a loop core (layers 6–17) and a coda (layers 18–23), and the core runs three times per token with shared weights. Each pass maintains its own KV cache and reads gated messages from the pass before it. The model therefore has 48 KV caches and 48 conv-state caches despite having only 24 layers.
vLLM's solution is to register each extra pass of a core layer as its own attention layer with its own KV and conv-state cache. The KV cache manager and prefix caching then treat these caches like any others, so serving the loop needed no new cache-management logic. This is the central engineering decision: the loop is absorbed into the existing layer abstraction rather than requiring a new execution path.
The cross-pass messages are the second challenge. From the second pass onward, each core sublayer input — the attention's query, key and value inputs and the MLP input — gets its own message, read from the previous pass's two blocks by an attention-residual read with a learned query per layer and input type. A rank-64 controller then merges the message into the input using a gated formula: the output is (1 − tanh(W_forget · h)) · input + tanh(W_gain · h) · message, where h is computed from RMS-normalized input and message via a down-projection. With 12 core layers and 4 inputs each, every later pass reads 48 messages per token.
A direct implementation would launch 4 message reads and 4 controllers, several kernels each, for every core layer in a later pass. vLLM instead reads all four messages of a layer in one attention-residual kernel and stacks the four controllers' weights, so their down and gain/forget projections run as batched GEMMs. One Triton kernel computes the controllers' RMS normalizations, and another fuses the gated merge with the RMSNorm of the sublayer the message feeds. The merge runs in FP32 without FMA contraction, matching the reference implementation's numerics.
For the layers themselves, vLLM reuses Inkling 2 Small's fused kernels — each sublayer's all-reduce, short convolution, residual update, attention-residual mix and RMSNorm fused into one kernel — generalized from hidden size 4,608 to 2,304, with a new mode that returns the FP32 mix before the RMSNorm so later passes can merge their messages into it.
Thinking Machines also released TML-d24-varloop-research, a checkpoint trained to run with a variable number of passes. vLLM exposes two options through --hf-overrides: loop_num_passes (at least 1, default 3) and loop_share_cache (all passes of a core layer share one KV cache, each overwriting the previous pass's keys and values; conv states stay per pass). On one GB200 at TP1 with 4K context and forward pass only, the measured decode-step times are 1.51 ms for 1 pass, 2.67 ms for 2 passes, and 3.85 ms for 3 passes at batch size 1; at batch 32, 3.81 ms, 5.92 ms and 8.11 ms respectively. KV cache capacity drops from 352K tokens at 1 pass to 175K at 3 passes. With 3 passes and a shared KV cache, decode time is essentially unchanged (3.86 ms at batch 1) but capacity recovers to 194K tokens — sharing frees less memory than dropping a pass because the per-pass conv-state caches remain.