Exa Agent Optimization
4 steps1 pieces3 Sources
AI search company Exa has developed a hierarchical agentic workflow to autonomously optimize inference kernels on AWS Neuron hardware, reporting a 47% improvement in embedding costs per million documents. The methodology uses AI agents to write custom NKI (Neuron Kernel Interface) kernels, validated by a strict 0.999 cosine similarity gate to prevent accuracy loss. This optimization was driven by the need to re-embed 30 billion documents during a model migration, which doubled their inference load.
The process involved three key steps: validating correctness against baseline datasets, measuring performance using the Neuron Profiler, and profiling to identify bottlenecks like memory spills and FLOPS waste. Agents handled tasks ranging from grid-searching compiler parameters to writing complex fused SwiGLU MLP kernels that eliminated high-bandwidth memory stalls. A shape-specific attention kernel further reduced padding waste, unlocking significant speedups.
This matters because unit economics at web scale are critical; small per-document cost improvements compound into massive savings. While Exa reports these gains as self-validated, the use of agentic coding for low-level kernel development represents a shift in how ML infrastructure is optimized. The approach allowed them to transfer optimizations directly between Inferentia2 and Trainium2 chips, enhancing flexibility across multi-region deployments.
How it got here
November 19, 2026: Exa publishes details on optimizing embedding throughput by 50% on AWS AI chips using an agentic development workflow.
aws.amazon.com ↗Prior to publication: Exa initiated a backfill process to re-embed 30 billion high-priority documents, doubling their embedding load and triggering hardware re-evaluation.