ReadingResearchRadarInvestment framework
Sign in / Sign up
← Back to reading
NewsMaterial6 min readDisconfirmAI Editorial

Tableau's Solutions Engineer and Market Strategy Lead: AI Agent Accuracy Depends More on Setup Than on the Model

Will Pitzler and Andrew Hill describe six configuration and governance practices that took custom Tableau MCP agents from prototype to production for a regulated enterprise customer, arguing that scoping, metadata preparation and evaluation discipline matter more than model choice.

2026-11-21 02:44 GMT+8

Will Pitzler, Senior Director of Market Strategy and Ecosystem at Tableau, and Andrew Hill, a Distinguished Solutions Engineer at Tableau, describe a customer engagement in which the team learned that an agent's accuracy depends far more on how it is configured, scoped and maintained than on which model it runs. Their customer, a large enterprise in a highly regulated industry, needed agents that could answer specific questions from Tableau data and act on those answers without leaving the agent environment. The stakes were high: in a regulated context, acting on a wrong answer can become a compliance issue, not just an inconvenience.

Tableau offers ready-to-use agents like Tableau Agent and Tableau Agent in Pulse, which are tightly scoped to improve accuracy out of the box. But this customer needed more: specialized workflows where accuracy was paramount, agentic workflows that let users act on findings without leaving the agent, embedding outside Tableau in a purpose-built application, fine-grained developer control over data access and tool permissions, and proof before production. This led them to build custom agents with Tableau MCP, which gives control that a ready-to-use agent cannot, but demands more rigorous testing, tighter governance and ongoing monitoring.

The first practice applies to any agent, ready-to-use or custom: prepare the data model. Any agent can only be as accurate as the data model it reads. The authors recommend putting business logic as close to the data as possible. If logic can be written as a calculation, write a calculated field so it gives the same answer each time rather than leaving it to agent interpretation. For example, Open Pipeline should be defined as IF NOT [Stage] IN ('Closed Won', 'Closed Lost') THEN [Opportunity Amount] END. Logic that every agent should know goes into Tableau Knowledge business preferences or field descriptions. Logic that applies to only one agent goes into that agent's system instructions. They also recommend removing field duplicates, describing fields with units and caveats, naming fields based on business logic rather than database names, hiding irrelevant fields like join keys, and certifying the data sources the agent is allowed to rely on.

Sign in / Sign up
ReadingResearchRadarInvestment framework

The second practice is to define the agent's role before any development begins. An open-ended agent pointed at a broad schema must both understand the question and find the data source that holds the answer. Removing the second step reduces the risk of a wrong answer and can speed up response time. The authors challenged their customer to answer four questions: What problem is the agent solving? What domain is it serving? What can and cannot it answer? How should it answer? The answers then decide which data and tools the agent gets, and form the core of its system prompt.

The third practice is the principle of least privilege, a long-standing security practice applied to agents. It is tempting to expand an agent's capabilities by leaving extra tools enabled and connecting additional data sources, but each added capability brings new failure modes. The authors removed every tool the agent did not need, starting with list-datasources, which lets an agent browse every data source on the site. Instead, they gave the agent the specific ID of the data source it should use in its system prompt. This improves accuracy because the agent does not risk querying a similar but incorrect source, improves speed and reduces cost because every tool the agent does not call saves a step and the token consumption that comes with it, and enforces governance because the agent can only reach approved data. They then enforced these limits within Tableau MCP itself, so a user cannot bypass them by pasting a different data source ID into the chat.

The fourth practice is optional and involves trade-offs. The authors noticed that even after the agent knew which data source to use, it still called the get-datasource-metadata tool at the start of each session, adding noticeable latency. To fix this, they pulled metadata from Tableau when the agent started, inserted it into the system prompt dynamically, and removed the get-datasource-metadata tool altogether. This meant the agent started every question already knowing the current fields, which reduced their response time by almost 50%. However, this approach has limits: if an agent has access to more than one or two data sources, it likely makes more sense to restore the metadata tool so it can resolve field-level information at runtime. The other trade-off is freshness: because metadata is loaded at agent start, a mid-session change to the data source leaves the agent working from a stale copy. For their customer, this risk was negligible because the data source was locked and modifications had to go through a governed process.

The fifth practice addresses repeated questions. Once the agents were in use, many of the same questions came up repeatedly. Each time, the agent had to rebuild the query, choosing appropriate fields, filters and calculations, which consumed unnecessary tokens and opened up another chance for error. For these known questions, the authors recommend not relying on the agent to build the query at all. A developer can write and test the query once and add it to the agent as a custom tool. When an employee asks a common question like "How many days off do I have left?", the agent calls that tool with the signed-in employee's ID, and the model only has to explain the result. They ultimately recommend against chasing 100% accuracy from an agent: if a question needs that level of certainty, like an FP&A figure that feeds a financial report, build a dashboard instead.

The sixth practice is building an agent evaluation and monitoring framework. Unlike a deterministic dashboard, an agent does not always give the same answer the same way each time. A small change such as rewording the system prompt, switching models, or adding a tool can fix one answer and quietly break another. The authors developed a question bank of user questions paired with correct answers, ran it against different agent configurations using an evaluation tool like LangSmith, and measured accuracy, latency and token usage for each run. For less deterministic questions, they used an "LLM as a judge" approach where a second model evaluated response quality. They versioned prompts and settings, ran the full question bank before any change went live, monitored response time, token usage, tool failures and usage frequency in production, and established a regular review process for adding new questions to the bank.

The authors summarize four principles: it starts with the data model; setup matters more than the model; narrow scope improves accuracy; and production agents need enterprise discipline like versioning, testing and ongoing monitoring. None of these is difficult on its own. The harder part is ongoing maintenance: updating field descriptions as data changes, adding new questions to the bank, and monitoring performance. They note that teams treating an agent as a product to maintain, not a project to ship, are the ones whose agents stay accurate. All practices and the 50% response time improvement are self-reported from a single customer engagement by Tableau's own product and engineering staff, and the article does not provide independent benchmarks or detailed methodology for reproducing these results.

Sources

  • salesforce.comSalesforce↗
Continue researching →