LesenForschungRadarAnlageframework
Anmelden / Registrieren
← Zurück zur Lektüre
NewsThema · AMD智能体推理调度Wesentlich3 Min. LesezeitDisconfirmAI Redaktion

AMD Demonstrates Trade-offs in Agent Inference Scheduling: Active Session Latency Drops 22%, New Request Wait Increases 49%

The ATOM team at Advanced Micro Devices (AMD) explains the three-layer service challenges of agent coding workloads, highlighting the scheduling strategy of PrefillDelayer: it improves P90 token inter-token latency by 22% under 16 concurrent sessions, at the cost of a 49% deterioration in median time-to-first-token for new requests, and explains why this strategy is only enabled within specific concurrency ranges.

2026-11-17 08:00 GMT+8

The traffic patterns of agent coding sessions differ fundamentally from those of standard chat requests. In a technical blog post about its ATOM inference engine, AMD described a typical case: one session involved 198 model calls, with call #48 sending 195.5K tokens, of which only 1.3K were new content. The corpus-wide median input length was 142,016 tokens, while the median output was just 444 tokens. This pattern—extremely long inputs, extremely short outputs, and highly shared prefixes—combined with the bursty parallelism of sub-agents and idle intervals caused by tool execution, creates service challenges that are entirely different from those posed by standard requests.

AMD categorizes these challenges into three layers: memory (maintaining reusable history), scheduling (balancing new requests against ongoing decoding), and compute (reducing per-step overhead in small batches). Among these, the trade-offs in the scheduling layer most intuitively reveal the inherent contradictions of agent services. ATOM’s PrefillDelayer allows a fixed number of decode steps to run immediately after each prefill completes, during which newly arrived requests are held and merged into a single prefill operation. Under 16 concurrent sessions, this strategy reduced P90 token inter-token latency from 31.6 milliseconds to 24.6 milliseconds—a 22% improvement—allowing active sessions to maintain a more stable output rhythm; however, the median time-to-first-token for new requests increased from 1,182 milliseconds to 1,766 milliseconds, a 49% deterioration.

Anmelden / Registrieren
LesenForschungRadarAnlageframework

This contradiction is sharper in agent workloads than in standard chats because the session tree structure amplifies latency. A parent agent waits for results from its sub-agents, meaning any delay on one branch postpones the completion of the entire task. Consequently, AMD enables PrefillDelayer only at concurrency levels of 16 and above, keeping it disabled in lower-concurrency interactive ranges, and provides different configuration combinations in deployment recommendations based on concurrency levels.

Optimizations in the memory and compute layers each have their own applicable conditions. ReplaySSM replaces the full state snapshot for every draft position in speculative decoding with a single checkpoint plus compact update records. In experiments with seven draft tokens, this freed up 24.49 GiB per GPU card, increasing available KV blocks from 8,325 to 18,813—a 2.26x increase—but this adds historical retention capacity rather than throughput or cache hit rates. Mono decode fuses multiple small kernels in individual KDA layers into two persistent kernels, reducing the target model's forward pass time from 22.0 milliseconds to 16.4 milliseconds in single-request tests; however, this optimization is currently enabled only at single concurrency, and its effects at higher concurrency levels remain unmeasured.

In terms of comprehensive effects, AMD reported that at 48 concurrent sessions, 94.1% of prompt tokens were reused rather than recomputed (with 77.3% coming from GPU HBM and 16.8% recovered from CPU DRAM), whereas the theoretical upper limit for this corpus is 97.2%. At 56 concurrent sessions, eight MI355X chips achieved a total of 17.5K tokens per second per chip. These figures come from AMD’s runs on the SemiAnalysis InferenceX AgentX benchmark and represent engineering references from the vendor under specific hardware and model combinations, not universally validated independent third-party performance benchmarks.

Quellen

  • rocm.blogs.amd.comROCm Blogs↗
Thema

AMD智能体推理调度

AMD ATOM团队针对智能体编码工作负载(AgentX)的长历史复用与突发子代理特性,提出了三层服务优化策略。核心在于PrefillDelayer调度机制:在16并发场景下,该策略通过延迟预填充请求以合并短批次,使活跃会话的P90 tok…

5 Schritte1 BeiträgeAktualisiert 2026-11-20 09:44

Wie es dazu kam

  1. 2026-05-14Red Hat与AMD宣布合作,推出支持AMD Instinct MI355X GPU的Red Hat AI 3.4,强调智能体AI需要新的基础设施栈,包括GPU用于复杂推理和CPU用于编排。redhat.com ↗
  2. 2026-07-21AMD发布技术文章,介绍与Moonshot AI合作重建的智能体AI服务栈,提出UMBP(统一内存与带宽池)概念,解决KV缓存跨层级调度问题。amd.com ↗
  3. 2026-09-03AMD发布PACE平台感知计算引擎,将其从纯推理引擎升级为智能体AI编排器,支持LangGraph作为原生编排层,实现确定性执行与基准测试。amd.com ↗

Was noch ungeklärt ist

  • PrefillDelayer策略在不同硬件配置或模型规模下的最优并发阈值是否一致?
  • UMBP多级缓存感知路由在实际生产环境中对总拥有成本(TCO)的具体影响数据尚未公开。
  • AMD PACE与ATOM/TokenSpeed等GPU加速引擎在混合部署时的协同调度效率如何?
Vollständiges Thema lesen →
Recherche fortsetzen →