LesenForschungRadarAnlageframework
Anmelden / Registrieren
← Zurück zur Lektüre
NewsThema · Koduri内存瓶颈论Wesentlich3 Min. LesezeitDisconfirmAI Redaktion

Raja Koduri, Who Led HBM Accelerator Development at AMD and Intel: AI Chip Competition Is Shifting from Compute to Data Movement

In SK Hynix's AI Ecosystem series, Raja Koduri—who led HBM accelerator development at AMD and Intel—argues that transformer architectures have undermined the caching assumption, gradually making the energy and latency costs of data movement a key bottleneck for AI system performance. Memory placement is therefore becoming the starting point of chip design.

2026-10-21 07:59 GMT+8

Raja Koduri made a judgment in the fourth installment of SK Hynix's AI Ecosystem series: the center of gravity in AI semiconductor competition is shifting from raw compute to data movement. He was involved in the development of an HBM-based GPU at AMD in 2015, and later led the development of Ponte Vecchio at Intel, which integrated 47 chiplets and four HBM2e stacks into a single package. These projects repeatedly confirmed a fact for him—signal integrity, power delivery, thermal management, and mechanical stability are the critical determinants of actual performance, not the floating-point throughput of a single chip.

His argument begins with physical constraints: a single multiplication or addition requires only a few femtojoules of energy, but reading data from off-chip DRAM can cost hundreds or even thousands of times more. The farther data must travel from the compute device, the greater the time and energy costs. For decades, computer architectures have relied on caches to amortize this cost—bringing data closer to the compute device and reusing it multiple times, spreading the initial retrieval cost across many operations. This worked well for conventional computing workloads.

But transformers have challenged this assumption. As model parameters have grown significantly, it has become difficult to keep all weights on-chip; during inference, each weight may be used only a limited number of times before new data is required, sharply reducing data reuse and making it more difficult for caches to sufficiently offset the cost of initial retrieval. As context length increases, the KV cache expands proportionally, while attention operations increase both computation and memory accesses. Systems must repeatedly read data from DRAM or HBM. Koduri notes that even optimization techniques such as FlashAttention, PagedAttention, and speculative decoding—which reduce the amount of data movement—cannot entirely eliminate the physical cost of moving data.

Anmelden / Registrieren
LesenForschungRadarAnlageframework

His conclusion: memory is becoming the starting point of AI architecture design, rather than an afterthought once computation is complete. HBM compresses the distance between compute and memory from centimeters to millimeters through vertical stacking and through-silicon vias; advanced packaging and hybrid bonding further shorten die-to-die connections; and processing-in-memory places some computation directly where the data resides, reducing energy consumption by 10 to 100 times for specific workloads. The competitive standard shifts accordingly—no longer whose chip has higher compute throughput, but who can simultaneously satisfy bandwidth, energy efficiency, reliability, and thermal management within a limited space.

Koduri distills the future competition into three questions: How close can data be placed to the compute device? When movement is unavoidable, how quickly and efficiently can data be transferred? And how much computational value can be extracted once data has been retrieved? He acknowledges that PIM faces severe thermal-management challenges—dissipating heat when logic circuits sit close to temperature-sensitive DRAM is extremely difficult, and bandwidth, power efficiency, reliability, and thermal performance must all be achieved simultaneously within a limited space. The concept is straightforward, but turning it into a practical product requires system-level co-design spanning memory, compute, packaging, thermal design, and software.

Quellen

  • news.skhynix.com↗
Thema

Koduri内存瓶颈论

前英特尔首席架构师、现Oxmiq Labs创始人Raja Koduri提出,AI芯片竞争的重心正从算力转向数据搬运。他在SK海力士AI生态系列文章中论证,Transformer架构打破了传统缓存复用假设,使得内存带宽和延迟成为系统性能的关键…

3 Schritte1 BeiträgeAktualisiert 2026-10-22 01:57

Wie es dazu kam

  1. 2024-02-17Raja Koduri在社交媒体上警告,若DRAM/SRAM每字节成本比低于5倍,芯片设计将转向SRAM,并因设计周期长导致后续难以回调。newsletter.bepresearch.com ↗
  2. 2026-09-29Koduri在TechSurge播客中阐述,AI基础设施瓶颈已从原始算力转移至内存层级,Oxmiq旨在通过高效数据移动提升Token生成速率。techsurgepodcast.com ↗
  3. 2026-10-21SK海力士发布Koduri撰写的AI生态系列文章第四篇,详细论证Transformer架构如何使数据搬运能耗和延迟成为AI系统性能的关键瓶颈。news.skhynix.com ↗

Was noch ungeklärt ist

  • DRAM与SRAM的成本比值是否已稳定在Koduri预测的临界点以下?
  • Oxmiq Labs的3D堆叠内存方案能否在量产中实现宣称的10倍带宽提升?
  • 随着Agentic AI的发展,内存瓶颈是否会进一步加剧现有GPU架构的局限性?
Vollständiges Thema lesen →
Recherche fortsetzen →