ReadingResearchRadarInvestment framework
Sign in / Sign up
← Back to reading
NewsQuick take2 min readDisconfirmAI Editorial

Lawyer's $9,450 private AI server with 12 used Nvidia V100s runs DeepSeek V4.1 Flash at 184 tokens/s

A lawyer upgraded his private AI server to run the full DeepSeek V4.1 Flash model on legacy hardware, achieving competitive speed and low cost.

2026-11-20

Nick Mermiges, a lawyer who builds private AI infrastructure, has upgraded his server from eight to twelve used Nvidia V100 GPUs for a total cost of $9,450. The system now runs every expert in the DeepSeek V4.1 Flash model using 4-bit NVFP4 quantization, achieving up to 184 tokens per second on code generation tasks.

The upgrade addresses a memory constraint: the original eight cards provided 254 GiB, but the model's experts require 278.9 GiB when loaded in NVFP4 format. Previously, Mermiges had to compress most experts to 2-bit precision. With the addition of a third board of four V100s, all 15,360 experts can now reside on the cards in their intended 4-bit precision without further compression.

A key technical limitation is that the Nvidia V100 architecture, released in 2018, lacks native FP4 or FP8 tensor cores. Consequently, every NVFP4 value must be unpacked into half-precision floating-point numbers via software before computation. The performance gain comes from reduced memory bandwidth usage—reading fewer bytes—rather than faster arithmetic operations.

Mermiges reports that the first token latency for a 256K context window dropped to 58 seconds, nearly twice as fast as his previous setup. Prompt reading speeds exceeded 4,500 tokens per second. He claims this configuration outperforms published stacks of three Nvidia DGX Spark units in single-request decoding while costing roughly half as much.

The system uses a specific topology where layers are divided across three boards, each containing four cards connected by NVLink. Hand-offs between the first two boards occur within a shared PCIe switch, but communication with the third board crosses host PCIe bridges, adding latency. Power caps were raised to 175 W per card, though pushing them to 200 W caused system instability during heavy multi-chat loads.

Sign in / Sign up
ReadingResearchRadarInvestment framework

Sources

  • tomshardware.comTom's Hardware↗
Continue researching →