Arize AI Engineer Xander Song Introduces Harbor: Evaluating AI Agents in Isolated Sandboxes
Xander Song, an engineer at Arize AI working on Phoenix, introduces Harbor, an open-source framework for evaluating AI agents using real data in isolated sandboxes. He explains concepts such as tasks, Docker environments, verifiers, rewards, and ATIF trajectories, and demonstrates how his team runs Harbor at scale on Daytona and Phoenix to test prompt changes affecting SQL tool usage.
Xander Song, an engineer at Arize AI who works on Phoenix, introduced the open-source framework Harbor in an ai.engineer podcast episode. The framework is designed to evaluate AI agents using real data within isolated sandboxes. Song detailed Harbor's core components, including task definitions, Docker environment configurations, verifiers and reward mechanisms, trial and job management, as well as oracles for validating setups before token consumption and ATIF trajectory formats.
The episode also demonstrated how Song's team runs Harbor at scale in parallel on Daytona and Phoenix platforms. A specific case study involved testing a prompt change that steered agents toward a new SQL tool, reducing tool calls without compromising correctness. For developers focused on AI agent infrastructure and evaluation methods, this resource offers practical insights into building reliable testing environments.