Micron held its Technology and Product Webinar last week. Chipstrat analyst Austin Lyons watched it and recorded his analysis, with the core judgment being: Micron gave an answer that was correct but missed the point regarding the memory demand question most concerning investors.
The starting point of the question came from analyst Vivek Arya. He asked a crucial question for memory manufacturers: today, memory accounts for more than half of an accelerator's bill of materials. Customers are discussing "despeccing," i.e., using less or cheaper memory. Could inference accelerators eventually be equipped with less DRAM or different types of DRAM? If so, Micron's bit sales would decline.
Micron's Scott DeBoer answered that system performance improves with increased memory capacity and bandwidth. Customers will choose the minimum specification that meets their product and business needs. Micron's job is to increase supply and drive bandwidth higher through products like vertical HBM. He concluded with: "Inherently, more memory does equal better system performance."
Lyons believes this answers the wrong question. DeBoer is right—more memory does help performance—but Arya didn't ask "is more memory better?"; he asked "what does it mean for memory suppliers when customers configure memory precisely according to workload?"
Lyons uses Google's chip design to illustrate that customers are already doing this. Publicly available specifications for Google Cloud's TPUs show: the inference chip TPU 8i has less compute but more memory; the training chip TPU 8t has more compute but less memory bandwidth because training is primarily compute-intensive. This is not hypothetical; it is differentiated products already in the market.
He further breaks down inference into two stages: prefill processes prompts and is compute-intensive; decode generates output token by token and is memory-intensive. Disaggregated inference runs these two stages on separate chips. This means prefill chips may not need as much HBM, while decode chips need more—and racks often contain more decode chips than prefill chips. Whether total memory demand decreases or offsets depends on the specific ratio.
Chipmakers are already differentiating memory configurations based on workload. AMD's MI455X carries 432GB of HBM4, and Nvidia's Rubin carries 288GB. Nvidia also announced the Rubin CPX, a prefill chip using 128GB of GDDR7 instead of HBM. Startup Positron's Asimov chip skips HBM entirely, using LPDDR5X with configurable capacities from 288GB to 2.3TB, allowing customers to choose capacity based on workload.
Here arises a dimension often overlooked: not just total volume, but structure (mix). GDDR7 and LPDDR are still DRAM, and Micron sells both. However, their profit margins differ from HBM. Lyons cites Micron's own data: HBM consumes three times the wafers per bit of DDR5, and "this trade ratio only increases with future generations of HBM." Therefore, every bit migrated from HBM to LPDDR or GDDR due to architectural optimization releases a significant amount of DRAM wafers.