"AI-Driven Data Foundation Design" Author Sanjeev Mohan: Agents Bypass Pipelines to Build Their Own Databases, Observability Must Be Embedded in Architecture
Data and AI consulting advisor Sanjeev Mohan stated at the KubeCon North America conference that AI agents, as a new class of data consumers, no longer wait for batch processing pipelines to complete; instead, they directly access databases, create short-lived instances, and parallelize tests across different reasoning chains. He argues that this runtime context construction requires observability to be an integral part of the architecture rather than an afterthought, predicting the emergence of new database types designed to support these needs.
For decades, enterprise data architecture followed a fixed path: data was generated once in business systems, then pushed through pipeline stages such as extraction, transformation, cataloging, and preparation before being consumed by humans for batch reports. Sanjeev Mohan believes the premise of this model is now being broken. His assessment is that when thousands of agents run simultaneously, they will not wait for pipelines to finish—they demand direct access to the database.
Mohan describes a behavioral pattern never before seen in software: agents create short-lived database instances, copy data within them, parallelize tests across different reasoning chains, and only then select a course of action. He cited a loan approval example: an agent might query census records and social media data simultaneously before rejecting an application. This immediately raises a governance question—how is that decision audited?
In his view, this differs fundamentally from past data governance. Past governance was deterministic: manage the data, then run a fixed reporting dashboard on top of it. Agent governance faces constantly changing context at runtime. For the same task, the context could be entirely different depending on the person (or the specific agent instance). Therefore, he stated that observability and transparency must become intrinsic parts of the architecture, rather than things bolted onto a data warehouse afterward.