AMD Releases InferaSim: Predicting LLM Serving Performance with Sparse Measurements to Cut Tuning Costs
AMD launches open-source tool InferaSim, which projects LLM serving configuration performance from limited real-world tests, saving over 95% of GPU trial time in internal evaluations.
US semiconductor company AMD released the open-source tool ROCm InferaSim on November 6, 2026, aiming to address the prohibitive costs of tuning Large Language Model (LLM) inference service configurations. Before deploying an LLM service, engineers must select parameters such as the inference engine, parallel layout, precision, and scheduler settings, generating thousands of candidate configurations. Traditional methods require hardware measurement for each configuration, with each trial tying up multi-GPU nodes for minutes. AMD stated that measuring every candidate in their search space would take approximately 25,000 GPU-hours, leading most teams to try only a few guesses and keep the best, while missed guesses result in extra GPU waste or failed latency targets.
The core idea behind InferaSim is "measure sparsely, transfer analytically." The tool separates parameters that change which kernels run (such as model, dtype, attention backend, graph mode, and speculative decoding) from those whose costs scale analytically (such as batch size, concurrency, context length, tensor-parallel width, and replica count). For the former, InferaSim performs one short measured run, called an "anchor," saved as a JSON file containing a batch sweep of measured prefill and decode step latencies plus a signature of what was running. For the latter, the tool performs fast analytical projections based on the anchor. Each projection takes under half a second on a single CPU core.
Internal tests on AMD's own Instinct MI355X GPUs showed that 11 anchors cost 9.1 GPU-hours and covered 226 configurations that took 204 GPU-hours to measure fully. This means GPU time is spent confirming a shortlist rather than exploring all possibilities. InferaSim can predict whether a recipe fits in HBM, the latency distribution under a given arrival process, the number of replicas needed for an SLO, and the best split between prefill and decode before any deployment exists.