AWS Team Self-Test: Three Strategies Reduce Token Consumption in AgentCore Large JSON Workloads by Up to 71%
When building a crew scheduling optimization agent for the airline operations platform Flightscape, the AWS engineering team faced average latencies exceeding 15 seconds with over 40 tools and megabyte-scale JSON inputs. They introduced three composable strategies—offloading heavy payloads to Amazon S3 while persisting only references, paginating oversized list results on demand, and replacing large dictionary outputs with structural descriptors—which reduced token consumption by up to 71% and average latency by up to 80% in self-tests using a representative 23 MB workload.
When production agents possess dozens of tools and return full structured data with each invocation, the context window fills within a few conversation turns. The AWS engineering team encountered this issue while building a crew scheduling optimization agent for the airline operations platform Flightscape: more than 40 tools, megabyte-scale JSON inputs, average request latencies exceeding 15 seconds, and some single tool responses exceeding 80,000 tokens. Every tool call injects the full payload into the model context, compounding pressure across multi-turn conversations, driving up costs, and degrading response quality.
Their first finding was that the bottleneck initially lay in the persistence layer. In the original design, heavy payloads such as optimization request JSON, response JSON, and validation reports were stored directly in session state, requiring serialization of these objects every turn. The team moved heavy payloads to Amazon S3, while AgentCore memory persisted only lightweight references (S3 key names). References are generated via content addressing—normalizing the JSON serialization and taking the SHA-256 digest—making writes idempotent: identical payloads map to the same key, allowing unchanged requests in subsequent turns to reuse existing objects without re-uploading.
After resolving the persistence layer, the bottleneck shifted to tool responses. Even with a lightweight session state, complete lists returned by tools were still injected verbatim into the model context. The team introduced on-demand paging: when tool results exceed a sampling threshold, the complete list is stored in S3, and the model receives only a compact summary (total count, initial samples, result handle). When more detail is needed, the model calls the paging tool to retrieve the required slice using offset and limit.