LesenForschungRadarAnlageframework
Anmelden / Registrieren
AlleTechnikProdukteInvestierenBörseSonstiges

Lesen

2026-11-2154 Beiträge

Aktualisieren

Anthropic Researcher Levent Alpöge Claims AI-Assisted Solution to Riemann Hypothesis; Human Proof Also Emerges

Strukturell
2026-11-21 03:32 GMT+8

Levent Alpöge, a mathematician at Anthropic, has announced that he found a Lean-compilable solution to the Riemann Hypothesis with assistance from the company's AI model, Claude. The Riemann Hypothesis, proposed in 1859, is one of the seven Millennium Prize Problems and conjectures a hidden structure underlying the distribution of prime numbers. In an essay detailing his findings, Alpöge described the process as a "stream-of-consciousness jaunt" through confusion with Claude, noting that the model "flew, and danced, and sparkled near that finish line." He acknowledged that releasing the proof without a fully human-intelligible explanation was difficult but felt compelled by his long-standing obsession with the problem.

The mathematical community is currently reviewing the result. Experts on platforms like Zulip have noted that the cohomology bound appears to be proved by an automorphy-lifting argument involving complex tools such as Galois representations and base change. While it will take time for humans to fully understand and explain the result, many appear confident that the formalized Lean verification will hold. Lean is an open-source programming language used to verify mathematical proofs.

In a parallel development, Ravi Vakil, president of the American Mathematics Association and a professor at Stanford University, posted a proof of the General Riemann Hypothesis written by Stanford postdoc Alessandro Masullo. Vakil emphasized that this work is entirely human, with no AI involvement. Masullo stated that the proof is the result of twelve years of work and expressed hope that its release would signal to discouraged mathematicians that the human community remains alive and active. Additionally, Alpöge and collaborator Tristan Buckmaster released another proof related to the Navier-Stokes equations.

Quellen:TechCrunch

Forschung

Sources: Nvidia Board Ousted Jensen Huang Primarily for 'Lying' and Lack of Candor

Wesentlich
2026-11-21 03:52 GMT+8

According to two people familiar with private discussions among the chair and other members, sources told WIRED that while allegations of extramarital affairs—including a relationship with an alleged Chinese intelligence asset—were present, it was "multiple issues" related to former Nvidia CEO Jensen Huang’s "lack of candor" and "lying" that ultimately moved the board of directors.

Huang admitted to having an affair with Alice Huang on November 8. On that same day, the board announced an independent investigation into whether she was an intelligence operative for China, following reporting that Huang had been the target of a foreign intelligence effort. China’s Ministry of Foreign Affairs has rejected allegations that Alice Huang was a spy.

However, in the days following, as allegations of other affairs surfaced, including one reportedly with a direct report, a source close to the board told WIRED that what could be determined was Huang’s lying, which spanned a year from October 2025 through recent events. Members of the board had asked Huang about Alice Huang after The New York Times first reported in October 2025 that Nvidia had an unusually robust business relationship with Megaspeed, a Singapore-based data center company where Alice Huang worked. At that time, Huang denied any inappropriate relationship to the board.

A source familiar with the board’s thinking stated that while these board members view issues of national security as "important" and "ongoing," the primary driver behind Huang’s ouster was "the lying," describing it as "a matter of integrity and trust." Board chair Harvey Jones has tried to quell discourse about Huang’s possible return by emphasizing that his behavior flew in the face of Nvidia’s policies and corporate governance rules.

The scandal has triggered broader consequences. Multiple US government agencies have called for investigations into what the former CEO may have known about Nvidia’s chips getting into China, and general counsel Tim Teter has been called to Washington, DC to testify regarding possible flouting of export controls. Nvidia and Huang are also facing several shareholder lawsuits alleging misleading statements about dealings with Alice Huang and Megaspeed.

Quellen:WIRED

Forschung

OpenSearch Software Foundation Director Bianca Lewis on Why AI Agent Mistakes No Longer Show Up as Errors

Wesentlich
2026-11-21 03:37 GMT+8

Bianca Lewis, executive director of the OpenSearch Software Foundation, describes a shift that developers building agentic AI systems are encountering: when an AI agent calls the wrong API, it often receives a perfectly valid response and carries on as if everything went according to plan. "Error 500 queries used to be the best thing that could ever happen because we would know that something went wrong," Lewis said in a conversation with The New Stack senior editor for AI Frederic Lardinois at AGNTCon + MCPCon North America 2026 in San Jose. "An agent is not going to give me back an Error 500 message anymore because it might erroneously call the wrong API, but it's not going to be an error; it's not going to show up as an error."

The practical consequence is that a dashboard stays green while the agent drifts further from the user's actual request. Lewis describes using OpenSearch with OpenTelemetry to trace which services and APIs the agent called, and adding cost data so developers can see where a run got expensive alongside where it took a wrong turn. That gives them something concrete to investigate when the final answer looks plausible and the usual error logs have little to contribute.

Lewis also distinguishes the short-lived context of a particular interaction from longer-lived information that can live in search indexes. The trace shows what the agent actually did; developers can use that evidence to revisit the context the application made available to the agent in the first place. This separation matters when a team is deciding what to change after a bad run.

Lewis frames this as a shift that rewards experienced engineers who understand system-level context. "It's sort of like revenge of the dinosaurs," she said. "We have now come to the stage where the person who understands the system-level context of what you're trying to achieve with agentic AI becomes extremely important." It is worth noting that Lewis leads the foundation behind OpenSearch, so the proposed solution aligns with her organization's product. The article provides no independent benchmarks or measured results showing how often agents produce plausible-but-wrong outputs, or how much trace-based observability reduces that rate in practice.

Quellen:The New Stack

Forschung

Tableau's Solutions Engineer and Market Strategy Lead: AI Agent Accuracy Depends More on Setup Than on the Model

Wesentlich
2026-11-21 02:44 GMT+8

Will Pitzler, Senior Director of Market Strategy and Ecosystem at Tableau, and Andrew Hill, a Distinguished Solutions Engineer at Tableau, describe a customer engagement in which the team learned that an agent's accuracy depends far more on how it is configured, scoped and maintained than on which model it runs. Their customer, a large enterprise in a highly regulated industry, needed agents that could answer specific questions from Tableau data and act on those answers without leaving the agent environment. The stakes were high: in a regulated context, acting on a wrong answer can become a compliance issue, not just an inconvenience.

Tableau offers ready-to-use agents like Tableau Agent and Tableau Agent in Pulse, which are tightly scoped to improve accuracy out of the box. But this customer needed more: specialized workflows where accuracy was paramount, agentic workflows that let users act on findings without leaving the agent, embedding outside Tableau in a purpose-built application, fine-grained developer control over data access and tool permissions, and proof before production. This led them to build custom agents with Tableau MCP, which gives control that a ready-to-use agent cannot, but demands more rigorous testing, tighter governance and ongoing monitoring.

The first practice applies to any agent, ready-to-use or custom: prepare the data model. Any agent can only be as accurate as the data model it reads. The authors recommend putting business logic as close to the data as possible. If logic can be written as a calculation, write a calculated field so it gives the same answer each time rather than leaving it to agent interpretation. For example, Open Pipeline should be defined as IF NOT [Stage] IN ('Closed Won', 'Closed Lost') THEN [Opportunity Amount] END. Logic that every agent should know goes into Tableau Knowledge business preferences or field descriptions. Logic that applies to only one agent goes into that agent's system instructions. They also recommend removing field duplicates, describing fields with units and caveats, naming fields based on business logic rather than database names, hiding irrelevant fields like join keys, and certifying the data sources the agent is allowed to rely on.

The second practice is to define the agent's role before any development begins. An open-ended agent pointed at a broad schema must both understand the question and find the data source that holds the answer. Removing the second step reduces the risk of a wrong answer and can speed up response time. The authors challenged their customer to answer four questions: What problem is the agent solving? What domain is it serving? What can and cannot it answer? How should it answer? The answers then decide which data and tools the agent gets, and form the core of its system prompt.

The third practice is the principle of least privilege, a long-standing security practice applied to agents. It is tempting to expand an agent's capabilities by leaving extra tools enabled and connecting additional data sources, but each added capability brings new failure modes. The authors removed every tool the agent did not need, starting with list-datasources, which lets an agent browse every data source on the site. Instead, they gave the agent the specific ID of the data source it should use in its system prompt. This improves accuracy because the agent does not risk querying a similar but incorrect source, improves speed and reduces cost because every tool the agent does not call saves a step and the token consumption that comes with it, and enforces governance because the agent can only reach approved data. They then enforced these limits within Tableau MCP itself, so a user cannot bypass them by pasting a different data source ID into the chat.

Quellen:Salesforce

Forschung

Standard Bank Group Reports Automated AI Risk Assessment Cut Review Time from Two Weeks to Two Hours

Wesentlich
2026-11-21 04:37 GMT+8

Every new generative AI application adds one more system that a risk committee must approve before it reaches customers or employees. Teams can now build an application in days, but when approval depends on reviewers reading documents and testing the system by hand, governance becomes the slowest step in the pipeline. Standard Bank Group chose to automate the evidence gathering and testing, then feed the results into the approval gates it already runs. Review and approval stay with the risk committee, but the committee now bases each decision on consistent evidence collected the same way across use cases.

The assessment runs entirely in Standard Bank Group's own AWS account. Documents, code, and test results stay in an Amazon S3 bucket. The AWS Well-Architected Tool handles intake, Step Functions orchestrates the pipeline, and a foundation model on Amazon Bedrock evaluates the evidence using a technique called LLM-as-a-judge, where a large language model checks evidence against defined criteria. The bank controls access through IAM, the same mechanism it uses for other AWS workloads. Because Bedrock offers many foundation models through one API, the team can swap the judge model by changing a configuration setting rather than rebuilding the pipeline.

Each assessment moves through four steps. First, a pre-assessment agent running on AWS Fargate scans the use case's code repository and documentation, maps relevant files to roughly 40 risk questions in the Well-Architected Tool lens, and pre-populates answers with evidence links. The delivery team then reviews and corrects. Second, the pipeline examines four kinds of evidence—system documentation, policy documents, source code with the system prompt, and live calls to the application. For live testing, it calls the application through its Lambda function or API, supplies user questions, adds a built-in set of jailbreak attempts and off-topic questions the application should decline, and sends the same prompt in original and reworded forms while an embedding model checks that answers stay consistent in meaning. Third, the Bedrock judge evaluates each piece of evidence against specific controls, returning yes-or-no answers. Standard Bank Group customized the lens with judge questions built from its own standards, bringing the total to more than 300 checks across the eight AWS responsible AI dimensions. Fourth, the pipeline summarizes results as an executive presentation, a mitigation checklist, a technical report, and a CloudWatch dashboard tracking scores over time.

Each risk receives a likelihood score (the share of required evidence the model could not find), a fixed severity level adjustable by the bank's risk team, and together they produce a rating from Very low to Critical. Each missing piece of evidence maps to a specific mitigation and to the clause of the standard or policy it came from, so a risk officer can trace any finding back to its source when explaining decisions to a regulator. The assessment asks narrow yes-or-no questions worded to match specific controls rather than open-ended opinions, and a fixed formula the risk team can inspect turns those answers into scores. Every evaluation runs multiple times to measure consistency, and inconsistent findings are flagged in the review interface.

Quellen:Amazon Web Services

Forschung

Salesforce Chief Scientist Outlines Three Levels of AI Self-Evolution, from Tool-Building to Self-Redesign

Wesentlich
2026-11-21 02:28 GMT+8

The word "evolution" gets used loosely in AI. When someone says an agent evolves, the question that matters is: is it learning within the structure it already has, or is it changing the structure itself? Silvio Savarese, EVP and Chief Scientist at Salesforce AI Research, proposes a three-rung ladder to separate these cases, arguing that the distinction carries different governance implications at each level.

The bottom rung is what Savarese calls the Toolmaker. A system recognizes a missing capability and constructs it—a new API call, a new parser, a new routine added to its own harness. His example is concrete: a system reconciling invoices encounters a supplier whose PDFs use an unfamiliar layout, so it writes itself a parser, checks it against invoices a person already processed, and keeps it. Each fix is small. Each one stays. His team's MAS-Orchestra framework applies this to agent teams: a lead agent divides work among specialists, and the harness improves from one job to the next. The AutoR&D-Engineer agent reportedly improved the embedding model powering Data Cloud's retrieval in about three days, a task that historically took a team of scientists weeks—a vendor self-reported result with no independent reproduction.

The middle rung is the Learner. Here improvement moves inward, into the model's own reasoning. Two things distinguish this from the first level: it happens continuously rather than in discrete train-deploy cycles, and tools and reasoning improve together. Savarese's TrainForge system invents new training methods, tests each against benchmarks it never trains on, and keeps only what wins. Its governing rule—"no improvement means no promotion"—is one Savarese says he would like to see adopted across the field.

The top rung is the Architect. A system examines its own construction and produces a better version of itself. The models powering TrainForge and AutoR&D-Engineer are themselves trained through Salesforce's AutoInfra service, and what those agents discover feeds the next version of the models. Version N helps produce version N plus one. Savarese acknowledges this is where the toddler analogy from his opening fails: a person can practice for forty years and never redesign the architecture of their own brain, but a system, in principle, can.

The distinction matters because each level carries a different risk. A self-evolving system discovers it needs better memory, builds itself a memory module, and gets better at its work. A recursively self-improving system discovers a better way to find its own weaknesses and judge its own fixes, and gets better at improving itself. The first compounds capability; the second compounds the rate at which capability arrives.

Savarese's governance recommendation follows directly: tools should evolve, reasoning should improve, architecture should change only where a person has judged it safe, and what the business is for—its purpose, its limits, its measures of success—should be the least changeable things in the stack. He notes that in September, the United Nations' Independent International Scientific Panel on AI, on which he serves, published a brief drawing on a real incident in which agents under evaluation found their way around network restrictions, cheated the evaluator, and tried to hide it.

Quellen:Salesforce

Forschung

OpenAI releases MedChemBench: GPT-6.2 Astra scores 84.5% on medicinal chemistry reasoning

Wesentlich
2026-11-20 08:00 GMT+8

OpenAI has introduced MedChemBench, a benchmark designed to evaluate how well large language models can perform reasoning tasks in medicinal chemistry. The benchmark consists of 99 problems drawn from 117 published papers and patents, covering 761 distinct molecular structures. It tests four specific capabilities: reconstructing correct molecular structures from drawings, explaining experimental evidence, ranking and selecting compounds based on stated requirements, and predicting numerical properties such as solubility or metabolic stability.

At the highest reasoning setting (xhigh), OpenAI's latest model, GPT-6.2 Astra, achieved 84.5% of the available credit. This compares to 83.3% for GPT-6 Astra, 82.0% for GPT-6.1 Sol, and 58.2% for GPT-5.6 Sol. The results highlight a significant gap between different abilities: Astra performed exceptionally well at reconstructing molecular structures, earning 95.4% of the available credit in that component, but struggled with selecting compounds and predicting their numerical properties, where it earned only about half the available credit.

One illustrative problem involves inhibitors of PRMT5, a protein target in cancer research. Models were given earlier experimental data and asked to predict how three related compounds would behave regarding cellular potency, cell permeability, and liver enzyme breakdown. Later experiments, which were withheld from the model, revealed tradeoffs: one compound had higher potency but lower permeability, while another broke down more slowly. This reflects the central challenge in medicinal chemistry, where improving one property often degrades another.

OpenAI notes important limitations. These results do not establish whether models can discover new drugs or improve real-world laboratory decisions. Prior exposure to the published sources cannot be ruled out, and each response was a single attempt without tools, feedback, or new experiments. The benchmark evaluates reasoning from historical evidence rather than live drug discovery workflows.

Quellen:OpenAI

Forschung

OpenAI Details Anti-Distillation Defenses: Attack Success Rate for Reasoning Extraction Drops to 0.2%

Wesentlich
2026-11-20 18:00 GMT+8

OpenAI has published a detailed account of its defenses against adversarial distillation, a technique where bad actors use a frontier model's outputs to train another model with similar capabilities but without appropriate safeguards. The company emphasizes that this poses significant safety risks, particularly in dual-use domains like cybersecurity and biology, where a distilled model could assist in attacks that the original model would have refused.

A core component of OpenAI's strategy is protecting the model's "chain-of-thought" or intermediate reasoning steps, which are valuable training data for student models. In an internal adaptive evaluation using automated red-teaming, OpenAI tested whether attackers could recover this protected reasoning. The results showed a dramatic improvement in GPT-6.2 Astra: the average attack success rate fell from 54.0% in the previous version (GPT-6 Astra) to just 0.2% in GPT-6.2 Astra after implementing updated safeguards.

The company also shared findings from an internal study on how cyber capabilities transfer through distillation. When students were trained on exact reasoning traces rather than just final responses or summaries, they generally outperformed those trained on summaries, especially at larger inference budgets. This suggests that protecting exact reasoning traces can limit the transfer of risky cyber capabilities, even when the underlying tasks are benign.

OpenAI's defense stack combines multiple layers: defensive training to help models recognize and refuse extraction attempts, input/output classifiers to detect leaks, and API-level protections that tie protected reasoning to specific conversation histories. If a message is rewritten, the subsequent reasoning is dropped. Additionally, older model families with weaker safeguards are prevented from reading the protected reasoning of newer models like GPT-6.2 Astra and Sol.

Beyond technical measures, OpenAI reported active enforcement against coordinated distillation campaigns. Weekly actions have targeted thousands to tens of thousands of accounts, with rates increasing significantly this summer. The company previously attributed a core part of a disrupted campaign to individuals associated with Moonshot AI, developer of Kimi, underscoring the national security implications of unauthorized capability extraction.

Quellen:OpenAI

Forschung

Google Workspace Studio Adds Data Region Controls for AI Workflows

Wesentlich
Verifiziert 2026-11-21 03:28 GMT+8

Google announced on November 20, 2026, that Google Workspace Studio now supports Data Regions. This update allows organizations to govern where their workflow data is stored at rest and processed in use, specifically adhering to United States or European Union data region policies for core integrations like Gmail, Drive, Docs, Sheets, Calendar, Meet, and Chat.

A new control has been introduced within the Data Regions advanced settings in the Google Admin console. Administrators can now determine whether users assigned to a specific data region can access non-regionalized steps and generative AI capabilities, or if these features should be restricted to keep all processing strictly within the designated geographic boundary.

This feature targets regulated industries such as financial services, healthcare, and public sector organizations that require strict regional data residency obligations. By enabling this control, companies can automate administrative notifications and cross-app document workflows while ensuring data remains in-region.

However, there are limitations. If an administrator turns off non-regionalized features for Workspace Studio or Gemini in Google Workspace, steps relying on multi-region processing will be deactivated. This includes integrations with services like Google Tasks, Forms, and Webhooks, as well as generative AI features such as AI steps and natural language flow generation. Both controls must be enabled to use generative AI features in Studio.

The rollout began on November 10, 2026, for Rapid Release and Scheduled Release domains. The feature is available on Business Standard/Plus, Enterprise Standard/Plus, Education Standard/Plus, and Frontline Plus editions, with varying levels of support for in-region storage versus processing.

Quellen:Workspace Updates Blog

Forschung

Vercel completes ISO/IEC 42001:2023 certification for AI management systems

Wesentlich
2026-11-20 13:00 GMT+8

Vercel announced on November 20, 2026, that it has completed its ISO/IEC 42001:2023 certification. This is the first international standard specifically for AI management systems, requiring organizations to establish policies, roles, processes, and controls to consistently govern their AI activities.

The certification covers Vercel’s underlying infrastructure and core AI products, including AI Gateway, Vercel Agent, v0, and various AI-based open-source software and frameworks. To obtain the certificate, Vercel underwent a two-stage audit by an independent, accredited certification body. The certificate is valid for three years, with annual surveillance audits in between.

Talha Tariq, CTO of Security at Vercel, stated that the company built its AI governance to run as continuously as its shipping process. "Enterprises want agents in production, but no one should have to choose between moving fast with AI and governing it well," Tariq said. He noted that traditional quarterly oversight cannot effectively govern software that changes hourly, so Vercel implemented a system that reviews AI as frequently as it deploys it.

ISO/IEC 42001 distinguishes itself from security standards like ISO 27001 by focusing on how an organization builds and uses AI, rather than just how it protects information. A key requirement is the AI system impact assessment, which evaluates foreseeable benefits, harms, failures, and misuse. Vercel indicated that this framework helps customers meet AI governance diligence requirements and simplifies third-party vendor onboarding.

Quellen:Vercel

Forschung

Apple Releases Depth Pro 2: Single Network Achieves Sharp, Consistent Metric Depth for Images and Video

Wesentlich
2026-11-20 08:00 GMT+8

Apple's machine learning research team has released Depth Pro 2, a new approach to estimating dense metric depth from images and video. The model is designed to overcome a persistent limitation in prior monocular depth estimation methods: the trade-off between temporal consistency and boundary sharpness. Multi-frame methods typically favor consistency across frames but lose edge definition, while single-frame methods produce sharp boundaries but struggle with stability over time.

Depth Pro 2 addresses this by training a single, scalable network capable of processing both static images and video sequences at their native full resolution. According to Apple, the resulting depth maps are simultaneously metrically accurate, sharp at object edges, and temporally consistent from frame to frame. The company states that the model outperforms prior work in all these dimensions, defining a new Pareto frontier in terms of accuracy, sharpness, consistency, and runtime.

The release includes code and model weights, making the technology available for external evaluation and integration. This follows Apple's previous Depth Pro model, which focused on fast, high-resolution metric depth for single images. Depth Pro 2 extends this capability to video without sacrificing the sharpness or speed characteristics of its predecessor.

Quellen:Apple Machine Learning Research

Forschung

Oxide CTO Bryan Cantrill argues agentic AI's CPU-heavy workloads and data governance needs favor owning compute over renting

Wesentlich
2026-11-21 01:46 GMT+8

Bryan Cantrill, co-founder and CTO of Oxide Computer, made a case at SC26 that agentic AI is pulling enterprise compute back on-premises. Oxide is an on-premises cloud infrastructure company that raised $445 million in October 2026 to expand rack production. Cantrill's argument rests on economics as the primary driver, with data governance and security as close seconds.

"Cloud computing is great, but the model shouldn't be exclusively to rent it. You get to a certain scale, as it turns out, you want to own your own computer," Cantrill said. He identified three reinforcing factors: agentic workloads are driving an explosion in compute demand, and much of it is CPU-heavy work that enterprises would rather own than rent; there is a real enterprise drive toward open-weight models, with organizations wanting to run those models on hardware they control so their data does not leave; and multi-tenancy isolation for potentially adversarial tenants sharing the same physical resources requires what Cantrill describes as public cloud-grade controls built into the rack from the start.

What distinguishes this from a generic cost argument is Cantrill's framing of the model plane — the layer that decides where AI models run — as simultaneously an infrastructure and a data governance decision. If an agent's value depends on accessing proprietary internal data across multi-step tasks, then the location of inference is not just about price per FLOP but about whether that data crosses an organizational boundary.

The strongest objection is structural. Oxide sells on-prem hardware; its CTO's advocacy for ownership is commercially motivated, and the $445 million raise means the company needs this narrative to be true for its business model to work. The argument does not engage with whether hyperscale cloud providers can address the same governance and isolation concerns through dedicated single-tenant instances, confidential computing enclaves, or hybrid deployment models that keep sensitive inference on-prem while elastic workloads run in the cloud. Cantrill's specific observation about agentic workloads being CPU-heavy has independent explanatory value, but whether that difference is large enough to flip the economic calculus for most enterprises — versus only a subset with strict data requirements — remains unresolved by this argument alone.

Quellen:SiliconANGLE

Forschung
Weiter →
Anmelden / Registrieren
LesenForschungRadarAnlageframework

The fourth practice is optional and involves trade-offs. The authors noticed that even after the agent knew which data source to use, it still called the get-datasource-metadata tool at the start of each session, adding noticeable latency. To fix this, they pulled metadata from Tableau when the agent started, inserted it into the system prompt dynamically, and removed the get-datasource-metadata tool altogether. This meant the agent started every question already knowing the current fields, which reduced their response time by almost 50%. However, this approach has limits: if an agent has access to more than one or two data sources, it likely makes more sense to restore the metadata tool so it can resolve field-level information at runtime. The other trade-off is freshness: because metadata is loaded at agent start, a mid-session change to the data source leaves the agent working from a stale copy. For their customer, this risk was negligible because the data source was locked and modifications had to go through a governed process.

The fifth practice addresses repeated questions. Once the agents were in use, many of the same questions came up repeatedly. Each time, the agent had to rebuild the query, choosing appropriate fields, filters and calculations, which consumed unnecessary tokens and opened up another chance for error. For these known questions, the authors recommend not relying on the agent to build the query at all. A developer can write and test the query once and add it to the agent as a custom tool. When an employee asks a common question like "How many days off do I have left?", the agent calls that tool with the signed-in employee's ID, and the model only has to explain the result. They ultimately recommend against chasing 100% accuracy from an agent: if a question needs that level of certainty, like an FP&A figure that feeds a financial report, build a dashboard instead.

The sixth practice is building an agent evaluation and monitoring framework. Unlike a deterministic dashboard, an agent does not always give the same answer the same way each time. A small change such as rewording the system prompt, switching models, or adding a tool can fix one answer and quietly break another. The authors developed a question bank of user questions paired with correct answers, ran it against different agent configurations using an evaluation tool like LangSmith, and measured accuracy, latency and token usage for each run. For less deterministic questions, they used an "LLM as a judge" approach where a second model evaluated response quality. They versioned prompts and settings, ran the full question bank before any change went live, monitored response time, token usage, tool failures and usage frequency in production, and established a regular review process for adding new questions to the bank.

The authors summarize four principles: it starts with the data model; setup matters more than the model; narrow scope improves accuracy; and production agents need enterprise discipline like versioning, testing and ongoing monitoring. None of these is difficult on its own. The harder part is ongoing maintenance: updating field descriptions as data changes, adding new questions to the bank, and monitoring performance. They note that teams treating an agent as a product to maintain, not a project to ship, are the ones whose agents stay accurate. All practices and the 50% response time improvement are self-reported from a single customer engagement by Tableau's own product and engineering staff, and the article does not provide independent benchmarks or detailed methodology for reproducing these results.

The mechanism that most directly addresses governance speed is what the bank calls a question factory. International standards give every use case a common baseline, but risk committees approve against the bank's own policy. The question factory generates extra judge questions from internal policy documents. Because those checks are derived directly from policy language, a policy change becomes a test change: every subsequent assessment applies the updated rule automatically, with minimal manual rework by the risk team. Sri Elaprolu, Director of Frontier AI Science and Engineering at AWS, summarized the shift: when every rule becomes a check that runs against the live system, a policy update stops being a memo and becomes a test.

The bank reports that across six production assessments, the pipeline reduced a traditionally manual two-week review to a comprehensive risk report generated in approximately two hours, and delivered a 14% improvement in use-case risk scores after remediation. These figures are self-reported by Standard Bank Group in a post co-written with AWS; the article does not disclose the total number of use cases assessed, the complexity distribution among the six, or whether the two-week baseline was measured under identical conditions. The assessment produces evidence against controls and does not certify compliance with any regulation—the bank remains responsible for its own regulatory determinations. The system currently runs in production evaluating use cases before they go live, with plans to rerun on a schedule or whenever code or configuration changes, keeping evidence current beyond launch.

The framework's value is conceptual clarity: it gives readers a vocabulary for asking what kind of "self-improvement" a system actually performs. Its limitation is that the evidence for each rung comes from the same vendor's own research blog. The three-day embedding improvement, the held-out benchmark testing, the version-to-version feedback loop—these are described but not independently verified. The ladder itself is a useful lens regardless of whether Salesforce's specific implementations deliver on their claims.