Microsoft's Maia 300 Architecture: Agentic Inference Shifts the Bottleneck from Compute to Data Movement

Rani Borkar, President of Azure Hardware Systems and Infrastructure, and Saurabh Dighe, Corporate Vice President for Systems Planning and Architecture, published a technical deep dive on October 12, 2026, systematically laying out the architectural logic behind Maia 300. Their central argument is that agentic workloads—including multi-step execution, long-context decoding, and tool use—have changed the computational profile and arithmetic intensity of inference. Simply adding power and silicon area is no longer sufficient to improve performance per watt; the architectural focus must shift to data locality, precision formats, and bandwidth optimization.

Maia 300 implements this shift through several concrete changes. Memory moves from a uniform access (UMA) organization to a non-uniform (NUMA) layout across chiplets, allowing software to place data near the compute that uses it. The system introduces enhanced MXFP4 and sub-3-bit storage formats to reduce bytes moved per token. The rack doubles to 96 accelerators, and the scale-up domain grows to a superpod of 12,288 accelerators.

Microsoft states that compared with Maia 200, Maia 300 delivers up to 7 times greater rack-scale token throughput in its full 96-accelerator configuration and more than 2.5 times the performance per watt. Memory bandwidth increases by 2.5 times, capacity by 33 percent, and HBM bandwidth per watt approximately doubles.