ReadingResearchRadarInvestment frameworkAI Company Primary Sources
Sign in / Sign up
← Back to reading
NewsMaterial3 min readDisconfirmAI Editorial

Startup Memory Race Condition in vLLM Multi-Instance GPU Sharing: Cisco DevNet Engineers Explain Why Two Flags Are Indispensable

The Cisco DevNet platform engineering team encountered a concurrent startup memory race condition when deploying three vLLM inference instances on a single NVIDIA A10G GPU. They explain how vLLM's default mechanism of allocating KV cache based on visible free memory at startup leads to overselling among multiple instances, and why using only the --kv-cache-memory flag to fix the cache size is insufficient—lowering the --gpu-memory-utilization target is also required, otherwise subsequent instances will fail to come online due to startup check failures.

2026-10-24 00:10 GMT+8

Time-slicing divides the GPU's compute time among multiple instances but does not partition video memory (VRAM). The Cisco DevNet platform engineering team configured AWS g5.2xlarge instances to run three time-sliced replicas per NVIDIA A10G (approximately 23 GiB VRAM) for their GPU learning labs. This means three learners share the same VRAM pool with no hardware isolation boundary between them. Since the A10G does not support MIG (Multi-Instance GPU), hardware-level partitioning is not an option.

The issue lies in vLLM's startup logic. Each vLLM instance determines its own KV cache size based on the free VRAM it sees at startup. When three instances start almost simultaneously, each believes it has exclusive access to most of the VRAM and requests a large chunk of cache independently—resulting in overselling. Typically, only the first instance starts successfully, while the other two report errors: "No available memory for the cache blocks." This phenomenon is documented in vLLM community issue #10643, but no complete solution for multi-instance coexistence was provided there.

The Cisco team's fix requires two flags working together because vLLM makes two separate memory decisions at startup. The first is --kv-cache-memory 3120851354, which fixes each instance's KV cache at 2.91 GiB, stopping dynamic measurement from free memory. Without measurement, there is no race—three instances no longer compete for the same unoccupied space.

Sign in / Sign up
ReadingResearchRadarInvestment frameworkAI Company Primary Sources

However, fixing the KV cache alone is not enough. vLLM performs a second check at startup: it verifies whether current free VRAM covers its utilization target (default 90%). Even with a fixed KV cache, this check remains. For example, with a 92% target, once the first instance has occupied VRAM, the second instance fails upon startup because the free VRAM (17.96/22.06 GiB) is less than required (20.29 GiB).

The second flag, --gpu-memory-utilization 0.30, resolves this specific check failure. Lowering the utilization target to 30% means each instance only needs to confirm that free VRAM covers approximately 6.6 GiB to start. After the three instances' KV caches total about 8.7 GiB plus model weights, the remaining VRAM still satisfies the 30% target, allowing later-starting instances to pass the check and coexist with already resident peers.

Each flag eliminates an independent failure path: --kv-cache-memory eliminates allocation race conditions, and --gpu-memory-utilization 0.30 eliminates startup check failures. Without either one, the three-instance solution cannot run stably. After configuration, all three vLLM engines start simultaneously and successfully complete the full experiment workflow.

This mechanism's applicability extends beyond Cisco's learning labs. Any scenario involving concurrent startup of multiple vLLM instances on the same GPU—such as multi-tenant inference services, parallel evaluation in CI pipelines, or shared development environments—will encounter the same race condition. Specific values need to be recalculated based on total GPU VRAM, number of instances, and model size, but the logic of "fixing cache to eliminate race conditions and lowering utilization targets to pass startup checks" is universal.

Sources

  • blogs.cisco.comCisco Blogs↗
Continue researching →