ReadingResearchRadarInvestment framework
Sign in / Sign up
← Back to reading
NewsMaterial4 min readDisconfirmAI Editorial

AWS Logistics Team: Adding New Datasets Requires Only a JSON File, AI Agent Auto-Discovery Without Redeployment

The AWS North America Sort Center team built a stateless MCP server on Amazon ECS Fargate to decouple the tool layer from the agent layer. Onboarding a new dataset requires submitting only a JSON metadata file describing the table structure, business semantics, and column-level meanings; the server automatically discovers and indexes it during the next pipeline sync, allowing all connected agents to query the dataset without writing integration code or redeploying.

2026-11-07 01:13 GMT+8

When enterprise AI Agents need access to data lakes, the most common practice is for each Agent to embed its own SQL generation logic, schema knowledge, and connection handling. When datasets change, every Agent must be patched individually; permissions are scattered with no unified audit point; onboarding a new table typically requires days of custom development. Under the pressure of managing thousands of datasets while multiple teams build their own AI assistants, the engineering team supporting Amazon’s North America Sort Center (NASC) network chose a different path: centralizing data access capabilities into a single stateless MCP server, letting Agents focus solely on reasoning and orchestration without worrying about how underlying data is accessed.

The core design decision of this server is to decouple the tool layer from the agent layer. Agents evolve rapidly, but data access patterns should outlive any single Agent implementation. The specific approach is as follows: each table corresponds to a JSON metadata file recording the database service name, table name, business description, column-level semantics (including type, natural language explanation, and sample values), load type, and refresh frequency. After this file is picked up by the pipeline, vector embeddings are generated using Amazon Titan Text Embeddings and indexed into Amazon Bedrock Knowledge Bases. Once indexing is complete, all connected Agents can discover and query that table without manually rebuilding indexes or restarting the server.

Sign in / Sign up
ReadingResearchRadarInvestment framework

There is an easily overlooked but actually critical field in the metadata file: business_description_for_the_table. It does not just write the business meaning of the table, but also carries engine-specific query hints. For example, the example in the text writes:

In Athena SQL, always use date() to wrap string literals when comparing against timestamp columns.

Such hints directly reduce the number of validation failures after SQL generation. Column-level sample values are equally important—if the Agent sees that the sample value for warehouse_id is FC-EAST-07 rather than East region facility, the generated WHERE clause will use the correct format.

The architecture is divided into four layers, each independently scalable: the client layer handles authentication and session state (Amazon Cognito validates JWTs, DynamoDB stores conversation history); the MCP server layer performs query analysis, tool selection, and result orchestration (FastMCP framework, driven by Amazon Bedrock); the tool layer connects to Athena and Redshift to execute SQL and to Knowledge Bases for semantic retrieval; the knowledge base refresh layer is responsible for discovering and indexing metadata files. Fargate tasks are stateless, with session state persisted in DynamoDB, so tasks can be replaced or scaled at any time without losing context.

The team also introduced an optional context mapping mechanism: specifying which tables a particular application should focus on. When context limits the table scope, the MCP server searches only those tables in the metadata catalog, reducing noise and improving SQL generation accuracy, while keeping end-to-end latency within the 15 to 20 seconds range tested by the team. When no context is specified, the Agent searches all indexed tables.

Security controls are implemented in layers: the MCP server runs in private subnets and is not directly exposed to the internet; only the client layer can route requests to it via a public-facing Application Load Balancer with AWS WAF. The Athena query tools enforce SELECT-only operations at the tool level, rejecting write operations before they reach the data lake. All tool calls are logged to Amazon CloudWatch in structured JSON, providing security teams with a unified audit point across all Agents and applications.

The transferable insight of this pattern is: when your number of Agents and datasets grow simultaneously, turning the onboarding of new data from a coding problem into a configuration problem significantly reduces marginal costs. But its applicability boundaries are equally clear—the entire implementation is deeply tied to the AWS service stack (ECS Fargate, Athena, Bedrock, DynamoDB); the accuracy of metadata files relies on manual maintenance; automatic handling mechanisms for description drift or expiration are listed in the text as future plans rather than implemented features; and the 15 to 20 second latency is a self-tested result by the team rather than an independent benchmark.

Sources

  • aws.amazon.comAmazon Web Services↗
Continue researching →