ReadingResearchRadarInvestment framework
Sign in / Sign up
← Back to reading
News3 min read·2026-10-21DisconfirmAI Editorial

Raja Koduri, Who Led HBM Accelerator Development at AMD and Intel: AI Chip Competition Is Shifting from Compute to Data Movement

In SK Hynix's AI Ecosystem series, Raja Koduri—who led HBM accelerator development at AMD and Intel—argues that transformer architectures have undermined the caching assumption, gradually making the energy and latency costs of data movement a key bottleneck for AI system performance. Memory placement is therefore becoming the starting point of chip design.

ImportanceMaterialEvidenceE3 inspectableWrite-upStandard

Original publication time 2026-10-21 07:59 GMT+8

Raja Koduri made a judgment in the fourth installment of SK Hynix's AI Ecosystem series: the center of gravity in AI semiconductor competition is shifting from raw compute to data movement. He was involved in the development of an HBM-based GPU at AMD in 2015, and later led the development of Ponte Vecchio at Intel, which integrated 47 chiplets and four HBM2e stacks into a single package. These projects repeatedly confirmed a fact for him—signal integrity, power delivery, thermal management, and mechanical stability are the critical determinants of actual performance, not the floating-point throughput of a single chip.

His argument begins with physical constraints: a single multiplication or addition requires only a few femtojoules of energy, but reading data from off-chip DRAM can cost hundreds or even thousands of times more. The farther data must travel from the compute device, the greater the time and energy costs. For decades, computer architectures have relied on caches to amortize this cost—bringing data closer to the compute device and reusing it multiple times, spreading the initial retrieval cost across many operations. This worked well for conventional computing workloads.

But transformers have challenged this assumption. As model parameters have grown significantly, it has become difficult to keep all weights on-chip; during inference, each weight may be used only a limited number of times before new data is required, sharply reducing data reuse and making it more difficult for caches to sufficiently offset the cost of initial retrieval. As context length increases, the KV cache expands proportionally, while attention operations increase both computation and memory accesses. Systems must repeatedly read data from DRAM or HBM. Koduri notes that even optimization techniques such as FlashAttention, PagedAttention, and speculative decoding—which reduce the amount of data movement—cannot entirely eliminate the physical cost of moving data.

Sign in / Sign up
ReadingResearchRadarInvestment framework

His conclusion: memory is becoming the starting point of AI architecture design, rather than an afterthought once computation is complete. HBM compresses the distance between compute and memory from centimeters to millimeters through vertical stacking and through-silicon vias; advanced packaging and hybrid bonding further shorten die-to-die connections; and processing-in-memory places some computation directly where the data resides, reducing energy consumption by 10 to 100 times for specific workloads. The competitive standard shifts accordingly—no longer whose chip has higher compute throughput, but who can simultaneously satisfy bandwidth, energy efficiency, reliability, and thermal management within a limited space.

Koduri distills the future competition into three questions: How close can data be placed to the compute device? When movement is unavoidable, how quickly and efficiently can data be transferred? And how much computational value can be extracted once data has been retrieved? He acknowledges that PIM faces severe thermal-management challenges—dissipating heat when logic circuits sit close to temperature-sensitive DRAM is extremely difficult, and bandwidth, power efficiency, reliability, and thermal performance must all be achieved simultaneously within a limited space. The concept is straightforward, but turning it into a practical product requires system-level co-design spanning memory, compute, packaging, thermal design, and software.

Sources

  • Source materialnews.skhynix.com↗
Continue researching →