ReadingResearchRadarInvestment framework
Sign in / Sign up
← Back to reading
NewsMaterial3 min readDisconfirmAI Editorial

AWS Team Self-Test: Three Strategies Reduce Token Consumption in AgentCore Large JSON Workloads by Up to 71%

When building a crew scheduling optimization agent for the airline operations platform Flightscape, the AWS engineering team faced average latencies exceeding 15 seconds with over 40 tools and megabyte-scale JSON inputs. They introduced three composable strategies—offloading heavy payloads to Amazon S3 while persisting only references, paginating oversized list results on demand, and replacing large dictionary outputs with structural descriptors—which reduced token consumption by up to 71% and average latency by up to 80% in self-tests using a representative 23 MB workload.

2026-11-18 06:44 GMT+8

When production agents possess dozens of tools and return full structured data with each invocation, the context window fills within a few conversation turns. The AWS engineering team encountered this issue while building a crew scheduling optimization agent for the airline operations platform Flightscape: more than 40 tools, megabyte-scale JSON inputs, average request latencies exceeding 15 seconds, and some single tool responses exceeding 80,000 tokens. Every tool call injects the full payload into the model context, compounding pressure across multi-turn conversations, driving up costs, and degrading response quality.

Their first finding was that the bottleneck initially lay in the persistence layer. In the original design, heavy payloads such as optimization request JSON, response JSON, and validation reports were stored directly in session state, requiring serialization of these objects every turn. The team moved heavy payloads to Amazon S3, while AgentCore memory persisted only lightweight references (S3 key names). References are generated via content addressing—normalizing the JSON serialization and taking the SHA-256 digest—making writes idempotent: identical payloads map to the same key, allowing unchanged requests in subsequent turns to reuse existing objects without re-uploading.

After resolving the persistence layer, the bottleneck shifted to tool responses. Even with a lightweight session state, complete lists returned by tools were still injected verbatim into the model context. The team introduced on-demand paging: when tool results exceed a sampling threshold, the complete list is stored in S3, and the model receives only a compact summary (total count, initial samples, result handle). When more detail is needed, the model calls the paging tool to retrieve the required slice using offset and limit.

Sign in / Sign up
ReadingResearchRadarInvestment framework

However, not all oversized outputs are lists. Some tools return deeply nested dictionaries containing numerous scalar values and nested structures; retaining them verbatim is too costly, but discarding them entirely deprives the model of reasoning cues. For dictionary outputs, the team designed descriptor-based compression: list fields retain only type, count, and the first five samples; nested dictionaries retain only key counts; scalar values are retained inline when useful. The full payload remains in S3, retrievable by the model on demand via the handle.

The three strategies are composable: Strategy 1 runs at the start and end of each turn, while Strategies 2 and 3 run each time a tool returns a result. In benchmark tests over five conversation turns, the team reported total token usage dropping from approximately 289,500 to approximately 82,900 (a reduction of about 71%), and average trace latency decreasing from 15.4 seconds to 3.1 seconds (a reduction of nearly 80%). Across 100 sample questions, total tokens fell from 23.6 million to 8.5 million (64%), and average latency dropped from 7,930 milliseconds to 2,461 milliseconds (69%). For the heaviest single query (listing valid crews across all validation categories), input tokens dropped from approximately 32,000 to approximately 8,000, and latency decreased from approximately 165 seconds to 9.9 seconds.

It should be noted that the above figures are self-test results from the AWS/Flightscape team on specific workloads: 23 MB of airline crew scheduling JSON, more than 40 tools, LangGraph orchestration, and Amazon Bedrock model inference. The article does not disclose the specific model versions and inference parameters of the baseline configuration, nor does it quantify the impact of increased pagination tool invocations on end-to-end latency. For agents with fewer tools, different payload shapes, or different model inference configurations, actual benefits require individual verification. The core transferable insight is that context pressure from large JSON loads is not resolved in one go; new bottlenecks are exposed with each layer of optimization, requiring the sequential stacking of corresponding compression or offloading strategies.

Sources

  • aws.amazon.comAmazon Web Services↗
Continue researching →