ReadingResearchRadarInvestment framework
Sign in / Sign up
← Back to reading
NewsMaterial4 min readDisconfirmAI Editorial

Together AI's Agent Evaluation Practice: Using Weaker Models to Find Gaps, Two-Sentence Document Fixes Raise GPU Cluster Task Pass Rate from 27% to 100%

The Together AI team built an agent evaluation system with three configurations: baseline, documentation MCP, and skill packages. Testing with weaker models like Claude Sonnet revealed specific issues such as incorrect links in skill documents and API error messages pointing to non-existent routes. By modifying just two sentences in the skill file, the GPU cluster pricing task went from a 4/15 pass rate for Sonnet to a 100% pass rate across all models. The team concludes that relying solely on strong models can mask platform gaps, and improving agent experience often requires only minor modifications.

2026-11-16

When testing whether coding agents could correctly query GPU cluster pricing, Hassan El Mghari and others at Together AI observed a surprising phenomenon: even with skill packages enabled, Claude Sonnet 4.6 passed only 4 out of 15 attempts. The failure mode was highly consistent—Sonnet treated unmarked sample output from the CLI reference documentation as live inventory, skipped the region lookup, and guessed the cheapest option. The problem lay in Together's own documentation and links: the skill file pointed to the marketing page for GPU clusters rather than the pricing table. After opening the marketing page, the agent saw no actual prices and had to guess.

The fix was unexpectedly simple: adding two sentences to the skill file instructed the agent to check /v1/compute/regions for real-time regions and instance types, and to open the pricing table for current rates. After this change, the same task achieved a 100% pass rate across all tested models. However, this discovery came to light thanks to an easily overlooked evaluation strategy: using both strong and weak models simultaneously. During the same period, Opus 4.8 passed 13 out of 14 attempts with the exact same configuration—it noticed that the region response lacked prices and continued searching until it found them in the documentation. If Together had tested only with Opus, this gap might have remained hidden.

Sign in / Sign up
ReadingResearchRadarInvestment framework

To address this, Together AI built an internal agent evaluation system. They first compiled a list of common tasks developers perform on the platform (such as batch inference, building voice agents, and managing auto-scaling), each essentially serving as a prompt for the agent. Then, for each task, they ran three configurations: Baseline (the agent has only default tools, simulating a standard developer workflow), Documentation MCP (direct connection to documentation via an MCP server), and Skill Packages (loading instructions and examples specifically written for the agent). Each task was run twice with three different models, generating hundreds of runs for a complete evaluation matrix. Scoring was handled by an LLM judge, which assigned one of three verdicts: Pass, Incomplete, or Fail.

The Together team emphasizes that fail and incomplete verdicts do not necessarily tell you where the problem lies—whether it is the agent, the API, or missing evidence; they simply indicate where to start investigating. They also examine passing runs because agents often manage to bypass confusing responses to complete tasks, and those detours themselves expose areas needing repair. The GPU cluster task also exposed an API-level issue: during a baseline run, Opus 5.5 received a 400 error when calling /v1/compute/instance-types. The error message suggested checking /v1/clusters/regions for a list of regions, but that route did not exist and returned a 404. Together's own error message sent the agent down a dead end, and human developers following the same prompt would have been stuck as well.

The second case involved restructuring the skill package. Together previously maintained 14 separate skills, divided by product surface, which overlapped with each other and could bloat context if multiple were loaded. After merging them into a single unified version, the initial evaluation passed only 54 out of 72 runs (75%), with costs doubling. Transcript analysis identified three problems: the router was too long, causing agents to read 57% more skill content; code examples referenced models that were no longer online; and example scripts contained minor bugs. After cutting the router to under 6,000 characters, replacing all examples with confirmed online models, and adding automated periodic live-testing, the unified skill package passed 70 out of 72 runs (97.2%), which is 7 percentage points higher than the original 14 skills.

From June to October, the pass rate for Documentation MCP rose from 67% to 96%, and for Skill Packages from 75% to 97%, while the Baseline only increased from 75% to 82%. The Together team notes this is not a controlled comparison—different snapshots used different models and tasks, and some improvements stem from underlying model advancements. However, they attribute the extra 20+ percentage points gained by guided configurations over the baseline to document and skill fixes revealed by the evaluations. Their core conclusion is: anything developers can do manually should be programmatically achievable via CLI or SDK; documentation, CLI help text, and API error messages are the first places agents look when stuck; and improving agent experience often requires only small changes, but without running evaluations, you may never find the right modification point.

Sources

  • together.ai↗
Continue researching →