ReadingResearchRadarInvestment framework
Sign in / Sign up
← Back to reading
NewsMaterial4 min readDisconfirmAI Editorial

NVIDIA Researcher Dhruv Nathawani: Constructing Search Puzzles from Wikidata Graph Paths to Train Models to Use Search Tools Instead of Answering from Memory

At the AI Engineer World's Fair, NVIDIA research scientist Dhruv Nathawani presented a synthetic data pipeline that extracts paths via random walks on the Wikidata entity graph, transforms those paths into puzzles requiring multi-step searches to solve, and then uses search tools to execute the full trajectory for supervised fine-tuning. He reported that after generating and filtering 50,000 seeds, 7,000 training records were retained, moving the model’s performance on the BrowseComp benchmark from 0% to 30%. He also acknowledged difficulties in verification: for a puzzle about the ownership of U.S. Steel, both the generator and verifier provided outdated answers.

2026-10-12 00:30 GMT+8

If a model can answer directly from its parametric memory, it has no reason to call a search tool. Teaching models to search is therefore not just a matter of training phase design, but of data design: you must first construct questions where searching is the only viable path. At the AI Engineer World's Fair in October 2026, NVIDIA research scientist Dhruv Nathawani used NeMo Data Designer to demonstrate this complete pipeline from question design to training data.

The core technique involves turning random walk paths through the Wikidata entity graph into puzzles. For example, a path connecting NVIDIA → Jensen → Oregon State University → Benton County → Thomas Hart Benton generates the puzzle: 'Which company was founded by the person who attended the university located in the county named after Thomas Hart Benton?' The answer lies within the chain of relationships; the model cannot hit the answer with a single search step. It must break down clues, retrieve information step-by-step, and compare constraints. Nathawani’s experience suggests keeping the walk length between four and eight hops: too short, and the answer is too direct; too long, and the path easily drifts to generic nodes like 'human' that lack distinctiveness.

The pipeline progresses through three notebooks. The first uses Wikipedia topic CSVs as seeds, paired with a difficulty sampler and Jinja templates to generate question-answer pairs, without any search behavior yet. The second integrates the Tavily search tool via the Model Context Protocol (MCP), allowing the model to issue real queries while answering and retaining retrieval results, transforming the training material from isolated answers into inspectable attempt trajectories. The third uses Wikidata paths as seeds, having the model generate puzzles based on entities in the path, which are then executed by a search agent to produce end-to-end trajectories containing queries, returned content, and final answers for supervised fine-tuning.

Sign in / Sign up
ReadingResearchRadarInvestment framework

Nathawani reported the output numbers for this pipeline: 50,000 seeds were filtered down to 7,000 high-quality end-to-end SFT records after generation and filtering. After training, the model moved from 0% to 30% on the BrowseComp benchmark. He noted that this is the combined effect of the entire pipeline and cannot be attributed to a single sampler, prompt template, or filter layer. On Text-to-SQL tasks, he reported that Nemotron Nano improved from 26% to 41% on the BIRD benchmark, but explicitly distinguished this as a non-agentic setup, making it incomparable to state-of-the-art scores for agentic systems.

Verification is the most honest difficulty in this pipeline. Nathawani cited a specific example: a puzzle asking 'Which country owns U.S. Steel?' The agent answered 'United States,' and the verifier judged it correct. Later, upon searching, Nathawani suspected the company might have been acquired by a Japanese firm—he had used the same flawed premise for both the generator and verifier. Their agreement did not mean the answer was correct. He further pointed out that a simple deterministic matching function yielded only about 24% match rate, but this low number itself cannot distinguish whether the answers were poor, the reference standard was inappropriate, or the matching function failed to recognize acceptable responses.

His solution was to split evaluation into multiple directions: model judgments handle large-scale initial screening; human review of the model judgments confirms whether the evaluation standards are being correctly applied; and real-time information retrieval checks if underlying facts have changed. He emphasized that passing a hundred checks only proves that these hundred issues do not exist; it does not prove there are no other problems.

For practitioners, the transferable key takeaway from this pipeline is not the specific tool choice (Tavily was just a demo; Exa or custom search tools can replace it), but the data design philosophy: prioritize diversity over quantity, use the cheapest correct column types (sampling with Python rather than LLMs), overproduce then filter, check a few records during preview before launching batches, and treat the data generation recipe itself as version-controlled code to share.

This pipeline is currently used internally at NVIDIA for training the search capabilities of the Nemotron series models. Nathawani also mentioned reinforcement learning variants: instead of pre-executing every puzzle to generate target trajectories, one could generate only the prompt dataset and have the model call tools in real time during training, scored by a reward function in NeMo Gym. However, he clearly stated that generating a general-purpose reward function remains an unsolved problem; the team is considering it but has not yet implemented it.

Sources

  • youtube.com↗
Continue researching →