Zain Asgar, Founder of AI Inference Orchestration Company Gimlet Labs: Why Specialized Accelerators Are Making a Comeback as General-Purpose Chips Stop Getting Faster Automatically
In an article for a16z, Zain Asgar, Co-founder and CEO of Gimlet Labs, draws a parallel to the 1990s when general-purpose microprocessors defeated specialized architectures. He argues that the end of Dennard scaling and the stagnation of cost-per-transistor have eroded the automatic performance growth advantage of general-purpose chips, allowing specialized accelerators to regain competitiveness in the inference era.
In 1990, Eugene Brooks of Lawrence Livermore National Laboratory asserted at a supercomputing conference that nothing could withstand the attack of 'killer micros'—where 'micro' referred to microprocessors, meaning general-purpose chips that performed well on most types of computation. They were also being sold by the millions per year into desktop PCs, delivering faster and faster performance each year.
The victory of general-purpose chips was not because specialized chips did anything wrong. Asgar points out that general-purpose microprocessors actually outperformed them even on the target workloads for which the specialized chips were designed. The ideas behind vector processors were later absorbed into CPUs as SSE and AVX instructions, and massively parallel designs influenced multi-core CPU and GPU architectures—the legacy of specialized companies lives on inside general-purpose chips, but the companies themselves disappeared.
This virtuous cycle was supported by three mutually reinforcing elements: generality drives volume—microprocessors entered desktop PCs with shipments of tens of millions, while specialized hardware had much smaller volumes because it ran only one type of workload; volume drives performance—huge shipments brought lower costs and more R&D funding, with single-processor performance growing over 50% annually between 1986 and 2003; and performance in turn reinforced generality—faster clock speeds unlocked applications that were previously unfeasible, such as dedicated video decoding cards being eliminated within two years after the appearance of the Pentium II.
The first cornerstone of Dennard scaling—the proportional reduction of voltage in smaller transistors—failed around 2005. Frequencies therefore stagnated, and multi-core became a compensatory measure, but not all applications can be efficiently parallelized. Around 2012, the cost per transistor also hit bottom: the growth in fab and process costs exceeded the growth in the number of transistors per wafer. The memory wall was named as early as 1995 and remains unsolved—David Patterson estimates that NVIDIA GPU compute power grew 80-fold from 2012 to 2022, while memory bandwidth grew only 17-fold, because compute scales with chip area, whereas bandwidth is limited by chip perimeter.
Asgar felt this shift earlier during his PhD studies in Mark Horowitz’s group at Stanford. His co-founder Omid Azizi, working with Horowitz, found that a 720p video encoder running on a general-purpose chip was about 500 times less energy-efficient than running it on a specialized ASIC. When the free lunch disappeared, the energy efficiency advantage of specialized chips became meaningful again.
GPUs themselves are survivors of specialized chips. The GeForce 256 in 1999 did one thing: rendering triangles for game graphics. But unlike vector processors and Lisp machines, it had a mainstream audience and shipments of millions. After frequency stopped scaling in 2005, chips had to shift from doing the same thing faster to doing more things simultaneously. The design of GPUs, with thousands of simple cores and extremely high memory bandwidth, happened to align with this shift, making them an ideal platform for large-scale parallel workloads ranging from early machine learning to cryptocurrency to high-performance computing.
Bill Dally of NVIDIA broke down the sources of roughly 1,000x improvement in single-GPU inference performance over ten years at Hot Chips 2023: about 16x came from simpler number formats (such as FP32 to BF16), 12.5x from new instructions, 2x from leveraging sparsity, and process improvements from 28nm to 5nm contributed only 2.5x. This means that process advancements, which drove industry performance growth for forty years, now contribute less than other factors. Asgar thus proposes an interesting reversal: perhaps we should now call GPUs general-purpose accelerators and CPUs specialized chips.
Back to the present, inference demand is exploding. Asgar cites data from a 2026 Goldman Sachs report: token demand will grow 24-fold from 2026 to 2030, while available power will grow only about 2-fold. Under conditions where physical infrastructure remains scarce, efficiency per watt becomes the most critical competitive dimension. Inference itself is a highly heterogeneous workload—the prefill and decode phases benefit from different hardware trade-offs. Selecting the correct chip architecture for each segment of the workload can bring significant performance improvements, even if underlying resources no longer scale automatically as they did in the past.
Asgar believes there are three key differences between today and the 1990s: it is much easier to write optimized code for new accelerators using AI agents; chip design infrastructure is more mature (most chip companies do not need to build their own fabs); and the scale and power constraints of inference workloads far exceed those of the past. His company, Gimlet Labs, splits inference workloads across different chips such as NVIDIA GPUs, LPUs, Google TPUs, Cerebras wafers, d-Matrix, Trainium, and AMD GPUs, claiming that heterogeneous splitting technology is 2 to 10 times faster than homogeneous stacks under the same power envelope.
This conclusion comes with a boundary acknowledged by the author himself: GPUs are following the path of CPUs in the past, directly internalizing the concept of inference-specific hardware—adding Tensor Cores in 2017, and MXFP4 and NVFP4 in 2024. If GPUs continue to absorb specialized features, the window for external specialized chips may close again. Asgar expects that the pendulum will eventually swing back to general-purpose architectures, but the era of specialized chips has arrived; even if general-purpose chips win again in the future, today's specialized hardware may become tomorrow's mainstream, with its contributions continuing to live on in general-purpose chips.
A note on conflicts of interest: Asgar is the Co-founder and CEO of Gimlet Labs, whose main business is heterogeneous inference orchestration. The data regarding the 2 to 10x performance improvement from splitting technology comes from Gimlet's own analysis (labeled 10/20/26) and is not independently verified by third parties. The article also includes recruitment and promotion information for expanding its inference cloud. These do not negate the logical value of the historical analogy and physical constraint arguments, but readers evaluating whether heterogeneous orchestration can truly deliver commercial advantages should view Asgar's optimistic judgment as a claim from a stakeholder.