ReadingResearchRadarInvestment framework
Sign in / Sign up
← Back to reading
NewsMaterial4 min readDisconfirmAI Editorial

Google Engineers: Model Architecture Locks in TPU Performance Ceiling Before Training

Two Google engineers published a TPU model co-design guide, using Ironwood benchmark data to demonstrate that design-stage decisions such as 256-dimension alignment and LatentMoE compression determine the upper limit of MFU; pure system tuning can raise the MFU of a 30B MoE from 12% to 23%, while a throughput gap of approximately two times exists between aligned and unaligned models.

2026-11-16

Gagik Amirkhanyan, Staff Software Engineer at Google, and Vaibhav Singh, Software Engineer at Google, published a TPU model co-design guide on the developer blog. The core argument is contained in a single sentence: performance tuning begins with model definition, and architectural choices determine the ceiling. They wrote at the beginning of the article that architectural choices made before the first training step set permanent ceilings on the model's FLOP utilization (MFU), cost, and inference latency. While this may sound like common sense, their Ironwood empirical data provides specific numbers.

The hardware basis for understanding this argument is the TPU's 256×256 MXU (Matrix Multiplication Unit). When a model's d_model, FFN width, attention projection, and vocabulary size are not multiples of 256, the MXU tiles cannot be fully filled, leaving the hardware waiting for data alignment. The most frequently overlooked dimension is head_dim: at 64 or 128, the QK matrix multiplication in flash-attention leaves at least half of the MXU's compute power idle. The authors recommend reshaping head_dim to 256, while proportionally reducing the number of Q and KV heads to maintain total projection width and logical KV cache bytes unchanged.

On the Qwen3-30B MoE, reshaping attention from 32 heads × 128 dimensions to 16 heads × 256 dimensions results in identical parameter counts and FLOPs, yet end-to-end training speed increases, with gains widening as context length grows. This is not software optimization, but hardware alignment. The authors further point out that 256-aligned MoE models (DeepSeek-V3, Qwen3-235B) achieve 608–630 BF16 TFLOP/s/chip on Ironwood, whereas the unaligned GPT-OSS-120B reaches only 330 TFLOP/s/chip, a gap approaching two times.

Sign in / Sign up
ReadingResearchRadarInvestment framework

The second design lever is LatentMoE. Its operational sequence is as follows: first, routing decisions are completed across the full d_model width; then, routed tokens are compressed from d_model to a smaller latent width ℓ (typically d/ℓ is about 2–4 times); expert computation is executed in the latent space; finally, the result is projected back to the full width. This simultaneously cuts two MoE bottlenecks—expert weight reads during small batches and all-to-all dispatch limited by throughput. On a 30B/128-expert MoE, a LatentMoE with ℓ=768 shortens on-device permute time by 2.8 times, improves end-to-end training throughput by 14% (322 vs 270 TFLOP/s/chip), and frees up approximately 25 GB/device of activation HBM when remat checkpoints are anchored on the compressed latent.

Expert granularity is equally important, but requires distinguishing between two mechanisms. A larger shared expert shifts work into dense, communication-free GEMMs, which can overlap with EP dispatch/combine during training and amortize weight reads across the decode batch during inference. Fewer, wider routed experts halve the dispatch traffic and double the GEMM tile fill rate per expert. On a 63B LatentMoE (ℓ=768, K/N=1/32, 512 TPU v7x chips, EP=128), halving (N,K) from (512,16) to (256,8) while doubling d_ff and shared expert width increased training throughput by 57% and maintained lower loss across 120.8B tokens.

However, the authors also provide clear boundary conditions. All Measured data comes from Google's own Ironwood profiling program; they advise readers to treat trends and rankings as transferable, but view specific percentages as configuration-specific. Pure system tuning (SparseCore offload, MXU-aligned tiles, remat tuning) can indeed raise the MFU of a 30B MoE from 12% to 23% without changing the model, but the ceiling of this improvement is still determined by dimension alignment at the design stage. If head_dim is 128, no amount of tuning will prevent at least half of the MXU's compute power from idling.

For teams designing new models or planning to migrate existing models to TPUs, the practical significance of this article lies in the fact that performance budgets are not post-training parameter-tuning problems, but design constraints defined at the architecture level. 256-alignment is a rule with near-zero quality risk; LatentMoE and expert granularity adjustments require ablation verification, but both should be completed before the first training step.

The authors also note that these principles extend to eighth-generation hardware such as TPU 8t and TPU 8i; the inference logic remains unchanged, though thresholds shift with the hardware. Both TPU 8t and TPU 8i add native FP4 support; TPU 8i also introduces Boardfly topology and Collectives Acceleration Engine, but the core mechanisms of 256-alignment and latent compression remain valid.

Sources

  • developers.googleblog.com↗
Continue researching →