Unblocked Founding Engineer Peter Werry: Validated Queries Can Still Return Semantically Irrelevant Results
At the 2026 AI Engineer World's Fair, Unblocked founding engineer Peter Werry demonstrated a document query engine that revealed an easily overlooked agent engineering problem: while error feedback causes models to avoid dangerous operations, validated retries can still return results that are semantically irrelevant to the user's intent. The fix is to provide the model with new capabilities (such as full-text search) rather than relying on additional retries.
Peter Werry posed a seemingly simple question to Unblocked’s document query engine during his demonstration: find pull requests (PRs) whose titles contain the word "authentication." The model first attempted to use a regular expression, which was rejected by the validator because regular expressions were disallowed by policy. After receiving this error feedback, the model switched to an exact title match. This attempt passed validation, but no PR had a title that exactly matched the string. Werry pointed out that the model had found an allowed operation but lost the search behavior the user actually wanted: passing validation only indicates that a query is permitted to execute, not that it expresses the user’s intent. The solution was not more retries, but adding full-text search capability to the engine—feedback helped the model avoid prohibited operations, but only the new capability enabled it to answer the original question.