ReadingResearchRadarInvestment framework
Sign in / Sign up
← Back to reading
NewsMaterial3 min readDisconfirmAI Editorial

Hamel Husain Tests Anthropic's Evaluation Plugin: Look at Data Before Building Evaluations

AI evaluation practitioner Hamel Husain and Isaac Flath tested Anthropic's newly released Claude Code evaluation plugin on real apartment leasing conversation data, finding three core flaws in its workflow: it prompts users to create evaluations before they have reviewed the data, the annotation interface lacks context, and individual evaluators bundle too many failure types. He distilled a transferable standard for evaluating tools from this experience: if an evaluation tool does not place data review at the center of the workflow, it is not worth using.

2026-09-30

Hamel Husain is a practitioner in the field of AI evaluation. In a livestream with Isaac Flath, he used Anthropic's newly released Claude Code evaluation plugin to process a set of real conversation records from an apartment leasing assistant. This plugin includes two commands, build_eval and hill-climb, which claim to help users build evaluations, check graders, and improve their applications against them. Hamel usually does not write reviews of evaluation tools because software changes so rapidly that reviews have a very short shelf life; however, as a first-party tool released by Anthropic (a model vendor), it is likely to influence how people approach evaluations, so he decided to try it.

The results were not ideal. The first issue appeared at the very beginning: after Claude browsed the conversations, it directly listed several potential failure points and asked the users to immediately select one to create an evaluation, recommending call-transfer rules. At this point, the two had not yet reviewed the conversations themselves, making it difficult to judge whether this was a truly prioritizable failure. Hamel believes the correct approach is to look at the data first, perform error analysis, and then decide which failures are worth writing evaluations for—the tool should assist in discovering issues rather than skipping understanding and demanding commitment directly.

The second issue was the annotation experience. Claude generated a Markdown file listing data related to call-transfer failures, asking users to browse and tell it which labels were wrong. This meant reading long conversations in an editor and switching back to the chat window to report corrections. Hamel found this absurd—since they were using a coding agent, it should have directly built an annotation application that made conversations easy to read and allowed feedback to be submitted in situ. They eventually asked Claude to build a web application to replace it.

Sign in / Sign up
ReadingResearchRadarInvestment framework
inputs.md

The third issue was that the scope of the evaluator was too broad. The call-transfer evaluator generated by the tool checked four different failures simultaneously: whether it asked for confirmation more than once or transferred without asking for confirmation (LLM-as-a-Judge); whether it said something between user consent and transfer (Code-based eval); whether it said something during or after the transfer (Code-based eval); and whether it vocalized tool mechanics such as "triggering" a transfer (Code-based eval). Hamel believes one should focus on one error at a time, or at least separate those requiring LLM judgment from those requiring code evaluation. Worse still, the tool explained the evaluation logic using descriptions like "protocol_ok is the headline. A case passes only if all four checks pass," which Hamel called "AI slop"—he would rather see the code or the judging prompt directly.

However, there were also positive aspects. Hamel acknowledged that the plugin's out-of-the-box ability to discover issues exceeded other automatic evaluation schemes he had seen; it identified issues with human handoff, formatting, and voice agents. He still believed that iteratively looking at data with an agent is better, but this was the strongest performance he had seen with a "one-shot" issue discovery method.

Hamel's conclusion was to hold off on using this plugin for now. He already has the Eval skills workflow organized by himself and Shreya, which is more flexible and less prescriptive. He later spoke with the plugin's author, who appreciated the feedback and stated they would update it, so he believes it is worth revisiting in the future. However, the principle he distilled applies to any evaluation tool: ensure the tool helps you understand the data before choosing evaluations and carefully inspect its judgments. He believes most of the evaluation workflow should take place in a web application rather than a chat window to reduce friction in data exploration and annotation. If an evaluation tool does not put looking at data at the center of the workflow, it is not worth using.

Sources

  • hamel.devHamel’s Blog↗
Continue researching →