AMD Demonstrates Trade-offs in Agent Inference Scheduling: Active Session Latency Drops 22%, New Request Wait Increases 49%
The traffic patterns of agent coding sessions differ fundamentally from those of standard chat requests. In a technical blog post about its ATOM inference engine, AMD described a typical case: one session involved 198 model calls, with call #48 sending 195.5K tokens, of which only 1.3K were new content. The corpus-wide median input length was 142,016 tokens, while the median output was just 444 tokens. This pattern—extremely long inputs, extremely short outputs, and highly shared prefixes—combined with the bursty parallelism of sub-agents and idle intervals caused by tool execution, creates service challenges that are entirely different from those posed by standard requests.
AMD categorizes these challenges into three layers: memory (maintaining reusable history), scheduling (balancing new requests against ongoing decoding), and compute (reducing per-step overhead in small batches). Among these, the trade-offs in the scheduling layer most intuitively reveal the inherent contradictions of agent services. ATOM’s PrefillDelayer allows a fixed number of decode steps to run immediately after each prefill completes, during which newly arrived requests are held and merged into a single prefill operation. Under 16 concurrent sessions, this strategy reduced P90 token inter-token latency from 31.6 milliseconds to 24.6 milliseconds—a 22% improvement—allowing active sessions to maintain a more stable output rhythm; however, the median time-to-first-token for new requests increased from 1,182 milliseconds to 1,766 milliseconds, a 49% deterioration.
This contradiction is sharper in agent workloads than in standard chats because the session tree structure amplifies latency. A parent agent waits for results from its sub-agents, meaning any delay on one branch postpones the completion of the entire task. Consequently, AMD enables PrefillDelayer only at concurrency levels of 16 and above, keeping it disabled in lower-concurrency interactive ranges, and provides different configuration combinations in deployment recommendations based on concurrency levels.
Optimizations in the memory and compute layers each have their own applicable conditions. ReplaySSM replaces the full state snapshot for every draft position in speculative decoding with a single checkpoint plus compact update records. In experiments with seven draft tokens, this freed up 24.49 GiB per GPU card, increasing available KV blocks from 8,325 to 18,813—a 2.26x increase—but this adds historical retention capacity rather than throughput or cache hit rates. Mono decode fuses multiple small kernels in individual KDA layers into two persistent kernels, reducing the target model's forward pass time from 22.0 milliseconds to 16.4 milliseconds in single-request tests; however, this optimization is currently enabled only at single concurrency, and its effects at higher concurrency levels remain unmeasured.
In terms of comprehensive effects, AMD reported that at 48 concurrent sessions, 94.1% of prompt tokens were reused rather than recomputed (with 77.3% coming from GPU HBM and 16.8% recovered from CPU DRAM), whereas the theoretical upper limit for this corpus is 97.2%. At 56 concurrent sessions, eight MI355X chips achieved a total of 17.5K tokens per second per chip. These figures come from AMD’s runs on the SemiAnalysis InferenceX AgentX benchmark and represent engineering references from the vendor under specific hardware and model combinations, not universally validated independent third-party performance benchmarks.
Sources:ROCm Blogs