LeituraPesquisaRadarFramework de investimento
Entrar / Cadastrar
← Voltar para a leitura
NotíciasTópico · vLLM多实例显存竞态Material3 min de leituraEditorial DisconfirmAI

Startup Memory Race Condition in vLLM Multi-Instance GPU Sharing: Cisco DevNet Engineers Explain Why Two Flags Are Indispensable

The Cisco DevNet platform engineering team encountered a concurrent startup memory race condition when deploying three vLLM inference instances on a single NVIDIA A10G GPU. They explain how vLLM's default mechanism of allocating KV cache based on visible free memory at startup leads to overselling among multiple instances, and why using only the --kv-cache-memory flag to fix the cache size is insufficient—lowering the --gpu-memory-utilization target is also required, otherwise subsequent instances will fail to come online due to startup check failures.

2026-10-24 00:10 GMT+8

Time-slicing divides the GPU's compute time among multiple instances but does not partition video memory (VRAM). The Cisco DevNet platform engineering team configured AWS g5.2xlarge instances to run three time-sliced replicas per NVIDIA A10G (approximately 23 GiB VRAM) for their GPU learning labs. This means three learners share the same VRAM pool with no hardware isolation boundary between them. Since the A10G does not support MIG (Multi-Instance GPU), hardware-level partitioning is not an option.

The issue lies in vLLM's startup logic. Each vLLM instance determines its own KV cache size based on the free VRAM it sees at startup. When three instances start almost simultaneously, each believes it has exclusive access to most of the VRAM and requests a large chunk of cache independently—resulting in overselling. Typically, only the first instance starts successfully, while the other two report errors: "No available memory for the cache blocks." This phenomenon is documented in vLLM community issue #10643, but no complete solution for multi-instance coexistence was provided there.

The Cisco team's fix requires two flags working together because vLLM makes two separate memory decisions at startup. The first is --kv-cache-memory 3120851354, which fixes each instance's KV cache at 2.91 GiB, stopping dynamic measurement from free memory. Without measurement, there is no race—three instances no longer compete for the same unoccupied space.

Entrar / Cadastrar
LeituraPesquisaRadarFramework de investimento

However, fixing the KV cache alone is not enough. vLLM performs a second check at startup: it verifies whether current free VRAM covers its utilization target (default 90%). Even with a fixed KV cache, this check remains. For example, with a 92% target, once the first instance has occupied VRAM, the second instance fails upon startup because the free VRAM (17.96/22.06 GiB) is less than required (20.29 GiB).

The second flag, --gpu-memory-utilization 0.30, resolves this specific check failure. Lowering the utilization target to 30% means each instance only needs to confirm that free VRAM covers approximately 6.6 GiB to start. After the three instances' KV caches total about 8.7 GiB plus model weights, the remaining VRAM still satisfies the 30% target, allowing later-starting instances to pass the check and coexist with already resident peers.

Each flag eliminates an independent failure path: --kv-cache-memory eliminates allocation race conditions, and --gpu-memory-utilization 0.30 eliminates startup check failures. Without either one, the three-instance solution cannot run stably. After configuration, all three vLLM engines start simultaneously and successfully complete the full experiment workflow.

This mechanism's applicability extends beyond Cisco's learning labs. Any scenario involving concurrent startup of multiple vLLM instances on the same GPU—such as multi-tenant inference services, parallel evaluation in CI pipelines, or shared development environments—will encounter the same race condition. Specific values need to be recalculated based on total GPU VRAM, number of instances, and model size, but the logic of "fixing cache to eliminate race conditions and lowering utilization targets to pass startup checks" is universal.

Fontes

  • blogs.cisco.comCisco Blogs↗
Tópico

vLLM多实例显存竞态

vLLM在同一GPU上并发启动多个推理实例时存在显存分配竞态:每个实例将其他进程正在加载的模型权重误判为自身的非Torch开销,导致KV缓存被严重压缩甚至计算为负值而启动失败。Cisco DevNet平台工程团队在NVIDIA A10G上部…

5 etapas1 peçasAtualizado 2026-11-02 09:11

Como chegamos aqui

  1. 2024-11-25GitHub用户brokenlander在issue #10643中首次公开报告:在NVIDIA GH200上以Docker容器并行运行两个vLLM实例(Llama-3.1-8B与Qwen2.5-7B,各设gpu-memory-utilization 0.3),第二个实例的non_torch_memory被高估至28.43 GiB,kv_cache_size算出−15.61 GiB并报错"No available memory for the cache blocks"。该issue次日被关闭为not planned。github.com ↗
  2. 2025-05-04用户wehos在#10643评论中定位到vllm/v1/worker/gpu_worker.py第202–211行:total_allocated_bytes取自torch.cuda.mem_get_info()的全卡已用量减去当前torch分配量,因此包含了同卡其他vLLM实例的权重占用,导致available_kv_cache_memory异常偏低或为负。github.com ↗
  3. 2026-03-03EmilHaase提交issue #35920,描述统一内存架构(Jetson Orin、DGX Spark等共享系统RAM与VRAM的设备)上的变体:Linux内核将.safetensors文件缓存进系统RAM造成全局可用内存骤降,vLLM将其误归为非Torch开销;并发启动三个<10B模型时互相蚕食快照,KV缓存可算出负字节或直接AssertionError崩溃。该issue于2026-07-28被标记stale后关闭为not planned。github.com ↗

O que ainda está indefinido

  • vLLM维护者是否接受#55827提出的NVML per-process + torch.cuda.memory_reserved()混合测量方案,还是继续要求用户显式指定--kv-cache-memory-bytes?
  • #35920(UMA页缓存误归因)与#55827(离散GPU全卡差值误归因)是否会合并为一个统一修复,还是各自独立处理?
  • Cisco DevNet博客所述'两个标志缺一不可'的具体参数组合(--kv-cache-memory数值与--gpu-memory-utilization阈值)是否在公开文档中有可复制的配置示例?
Ler o tópico completo →
Continuar pesquisando →