AI Search Company Exa Optimizes Neuron Chip Inference Kernels with Hierarchical Agent Workflows, Reports 47% Improvement in Embedding Costs
Exa describes a methodology for AI agents to autonomously write custom NKI kernels to optimize embedding inference: a three-part framework (validating correctness, measuring performance, profiling to find directions) combined with an agent strategy layered by task complexity. Using a 0.999 cosine similarity gate prevents optimizations from introducing errors, resulting in a self-reported 47% improvement in cost per million documents embedded.
At the scale of billions of documents, the unit economics of embedding determine everything. Exa embeds web documents into vector databases for search and agent invocation; any improvement in cost per document multiplied by tens of billions results in significant savings. They selected AWS Neuron hardware (Inferentia2 and Trainium2) for inference, but the initial data-parallel FP16 implementation failed to fully leverage chip performance. Starting from naive compilation, a series of deep kernel optimizations was required to approach the hardware's theoretical throughput.
The optimization path itself is not uncommon: aligning weight layouts to eliminate transposes, using compiler autocast to reduce precision, pre-building attention kernels via NKI, fusing SwiGLU MLP kernels with custom weight stationarity to eliminate High-Bandwidth Memory (HBM) spills, and finally rewriting kernels tailored to specific model attention head shapes to cut FLOPS wasted on padding. These steps pushed throughput to a self-reported 1.5x increase. What truly warrants attention is not these individual steps, but the methodology driving them.
Exa describes a three-part agent framework: validating correctness, measuring performance, and profiling workloads to identify the next optimization direction. Each agent runs an independent workflow via Flyte—compiling NEFF executables, performing forward passes on baseline datasets, and requiring results to achieve 0.999 cosine similarity to pass. It then conducts a 30-minute saturation benchmark on real hardware to measure performance. This framework turns the optimization loop into an automatically verifiable closed system, ensuring agents cannot trade correctness for speed.