NVIDIA Researcher Dhruv Nathawani: Constructing Search Puzzles from Wikidata Graph Paths to Train Models to Use Search Tools Instead of Answering from Memory
If a model can answer directly from its parametric memory, it has no reason to call a search tool. Teaching models to search is therefore not just a matter of training phase design, but of data design: you must first construct questions where searching is the only viable path. At the AI Engineer World's Fair in October 2026, NVIDIA research scientist Dhruv Nathawani used NeMo Data Designer to demonstrate this complete pipeline from question design to training data.
The core technique involves turning random walk paths through the Wikidata entity graph into puzzles. For example, a path connecting NVIDIA → Jensen → Oregon State University → Benton County → Thomas Hart Benton generates the puzzle: 'Which company was founded by the person who attended the university located in the county named after Thomas Hart Benton?' The answer lies within the chain of relationships; the model cannot hit the answer with a single search step. It must break down clues, retrieve information step-by-step, and compare constraints. Nathawani’s experience suggests keeping the walk length between four and eight hops: too short, and the answer is too direct; too long, and the path easily drifts to generic nodes like 'human' that lack distinctiveness.
The pipeline progresses through three notebooks. The first uses Wikipedia topic CSVs as seeds, paired with a difficulty sampler and Jinja templates to generate question-answer pairs, without any search behavior yet. The second integrates the Tavily search tool via the Model Context Protocol (MCP), allowing the model to issue real queries while answering and retaining retrieval results, transforming the training material from isolated answers into inspectable attempt trajectories. The third uses Wikidata paths as seeds, having the model generate puzzles based on entities in the path, which are then executed by a search agent to produce end-to-end trajectories containing queries, returned content, and final answers for supervised fine-tuning.
Nathawani reported the output numbers for this pipeline: 50,000 seeds were filtered down to 7,000 high-quality end-to-end SFT records after generation and filtering. After training, the model moved from 0% to 30% on the BrowseComp benchmark. He noted that this is the combined effect of the entire pipeline and cannot be attributed to a single sampler, prompt template, or filter layer. On Text-to-SQL tasks, he reported that Nemotron Nano improved from 26% to 41% on the BIRD benchmark, but explicitly distinguished this as a non-agentic setup, making it incomparable to state-of-the-art scores for agentic systems.
Verification is the most honest difficulty in this pipeline. Nathawani cited a specific example: a puzzle asking 'Which country owns U.S. Steel?' The agent answered 'United States,' and the verifier judged it correct. Later, upon searching, Nathawani suspected the company might have been acquired by a Japanese firm—he had used the same flawed premise for both the generator and verifier. Their agreement did not mean the answer was correct. He further pointed out that a simple deterministic matching function yielded only about 24% match rate, but this low number itself cannot distinguish whether the answers were poor, the reference standard was inappropriate, or the matching function failed to recognize acceptable responses.
Sources:youtube.com