Google Engineers: Model Architecture Locks in TPU Performance Ceiling Before Training
Two Google engineers published a TPU model co-design guide, using Ironwood benchmark data to demonstrate that design-stage decisions such as 256-dimension alignment and LatentMoE compression determine the upper limit of MFU; pure system tuning can raise the MFU of a 30B MoE from 12% to 23%, while a throughput gap of approximately two times exists between aligned and unaligned models.
Gagik Amirkhanyan, Staff Software Engineer at Google, and Vaibhav Singh, Software Engineer at Google, published a TPU model co-design guide on the developer blog. The core argument is contained in a single sentence: performance tuning begins with model definition, and architectural choices determine the ceiling. They wrote at the beginning of the article that architectural choices made before the first training step set permanent ceilings on the model's FLOP utilization (MFU), cost, and inference latency. While this may sound like common sense, their Ironwood empirical data provides specific numbers.
The hardware basis for understanding this argument is the TPU's 256×256 MXU (Matrix Multiplication Unit). When a model's d_model, FFN width, attention projection, and vocabulary size are not multiples of 256, the MXU tiles cannot be fully filled, leaving the hardware waiting for data alignment. The most frequently overlooked dimension is head_dim: at 64 or 128, the QK matrix multiplication in flash-attention leaves at least half of the MXU's compute power idle. The authors recommend reshaping head_dim to 256, while proportionally reducing the number of Q and KV heads to maintain total projection width and logical KV cache bytes unchanged.
On the Qwen3-30B MoE, reshaping attention from 32 heads × 128 dimensions to 16 heads × 256 dimensions results in identical parameter counts and FLOPs, yet end-to-end training speed increases, with gains widening as context length grows. This is not software optimization, but hardware alignment. The authors further point out that 256-aligned MoE models (DeepSeek-V3, Qwen3-235B) achieve 608–630 BF16 TFLOP/s/chip on Ironwood, whereas the unaligned GPT-OSS-120B reaches only 330 TFLOP/s/chip, a gap approaching two times.