ReadingResearchRadarInvestment framework
Sign in / Sign up
← Back to reading
NewsMaterial3 min readDisconfirmAI Editorial

AI Search Company Exa Optimizes Neuron Chip Inference Kernels with Hierarchical Agent Workflows, Reports 47% Improvement in Embedding Costs

Exa describes a methodology for AI agents to autonomously write custom NKI kernels to optimize embedding inference: a three-part framework (validating correctness, measuring performance, profiling to find directions) combined with an agent strategy layered by task complexity. Using a 0.999 cosine similarity gate prevents optimizations from introducing errors, resulting in a self-reported 47% improvement in cost per million documents embedded.

2026-11-19 07:58 GMT+8

At the scale of billions of documents, the unit economics of embedding determine everything. Exa embeds web documents into vector databases for search and agent invocation; any improvement in cost per document multiplied by tens of billions results in significant savings. They selected AWS Neuron hardware (Inferentia2 and Trainium2) for inference, but the initial data-parallel FP16 implementation failed to fully leverage chip performance. Starting from naive compilation, a series of deep kernel optimizations was required to approach the hardware's theoretical throughput.

The optimization path itself is not uncommon: aligning weight layouts to eliminate transposes, using compiler autocast to reduce precision, pre-building attention kernels via NKI, fusing SwiGLU MLP kernels with custom weight stationarity to eliminate High-Bandwidth Memory (HBM) spills, and finally rewriting kernels tailored to specific model attention head shapes to cut FLOPS wasted on padding. These steps pushed throughput to a self-reported 1.5x increase. What truly warrants attention is not these individual steps, but the methodology driving them.

Exa describes a three-part agent framework: validating correctness, measuring performance, and profiling workloads to identify the next optimization direction. Each agent runs an independent workflow via Flyte—compiling NEFF executables, performing forward passes on baseline datasets, and requiring results to achieve 0.999 cosine similarity to pass. It then conducts a 30-minute saturation benchmark on real hardware to measure performance. This framework turns the optimization loop into an automatically verifiable closed system, ensuring agents cannot trade correctness for speed.

Sign in / Sign up
ReadingResearchRadarInvestment framework

A hierarchical agent strategy is the core design of this methodology. Lightweight models perform grid searches on compiler parameters—work that is voluminous but highly fault-tolerant, with correctness gates providing a natural safety net. Reasoning agents test complex hypotheses and locate bottlenecks, such as determining that per-accelerator throughput is significantly higher when one core is idle compared to both running simultaneously, thereby inferring inter-core contention. The strongest code generators write custom kernels to fix bottlenecks, such as rewriting a fused SwiGLU MLP from a pre-built implementation to a weight-stationary version. Matching different model capabilities to tasks of varying difficulty controls costs and prevents powerful agents from wasting compute on simple searches.

Because many large models are unaware of the latest features and syntax of the Neuron SDK, Exa also utilized specialized 'neuron agentic development skills' to supplement context. This highlights a practical constraint: the effectiveness of agent-assisted kernel optimization partly depends on whether the agent can access the target platform's latest interface documentation and best practices. General-purpose code generation capabilities require additional knowledge injection when applied to highly specialized hardware stacks.

For any team performing kernel optimization on custom accelerators, this structure provides a reusable template: breaking optimization into four stages—finding directions, writing code, verifying correctness, and measuring performance; matching models of different capabilities to different stages; and using strict numerical gates to prevent silent errors introduced by optimization. Compared to purely human tuning, it explores more hypotheses in parallel within the same time window. Compared to letting the strongest model handle everything, it delegates simple searches to cheaper models and concentrates compute power on the kernel writing that truly requires creativity.

The boundaries are equally clear. The 47% figure is a self-reported result from Exa under specific conditions: a specific embedding model, a specific workload, and a specific hardware instance (inf2.xlarge single NeuronCore). The specific steps in the optimization path are highly dependent on model architecture—SwiGLU fusion, customized attention shapes, and padding reduction were all done specifically for the characteristics of their model. The original text does not discuss whether the agent method can be migrated to models with different numerical tolerances (such as medical or financial scenarios requiring higher precision) or platforms lacking equivalent profiling tools. Furthermore, no analysis is provided on whether a 30-minute saturation benchmark is sufficient to capture all patterns of performance degradation.

Sources

  • aws.amazon.comAmazon Web Services↗
Continue researching →