AMD Demonstrates Trade-offs in Agent Inference Scheduling: Active Session Latency Drops 22%, New Request Wait Increases 49%
The ATOM team at Advanced Micro Devices (AMD) explains the three-layer service challenges of agent coding workloads, highlighting the scheduling strategy of PrefillDelayer: it improves P90 token inter-token latency by 22% under 16 concurrent sessions, at the cost of a 49% deterioration in median time-to-first-token for new requests, and explains why this strategy is only enabled within specific concurrency ranges.
The traffic patterns of agent coding sessions differ fundamentally from those of standard chat requests. In a technical blog post about its ATOM inference engine, AMD described a typical case: one session involved 198 model calls, with call #48 sending 195.5K tokens, of which only 1.3K were new content. The corpus-wide median input length was 142,016 tokens, while the median output was just 444 tokens. This pattern—extremely long inputs, extremely short outputs, and highly shared prefixes—combined with the bursty parallelism of sub-agents and idle intervals caused by tool execution, creates service challenges that are entirely different from those posed by standard requests.
AMD categorizes these challenges into three layers: memory (maintaining reusable history), scheduling (balancing new requests against ongoing decoding), and compute (reducing per-step overhead in small batches). Among these, the trade-offs in the scheduling layer most intuitively reveal the inherent contradictions of agent services. ATOM’s PrefillDelayer allows a fixed number of decode steps to run immediately after each prefill completes, during which newly arrived requests are held and merged into a single prefill operation. Under 16 concurrent sessions, this strategy reduced P90 token inter-token latency from 31.6 milliseconds to 24.6 milliseconds—a 22% improvement—allowing active sessions to maintain a more stable output rhythm; however, the median time-to-first-token for new requests increased from 1,182 milliseconds to 1,766 milliseconds, a 49% deterioration.