Reading

According to a report by TechNews, New York-based AI infrastructure startup Fragment announced the completion of a $12 million funding round, co-led by FirstMark and BoxGroup. The company focuses on providing a security control layer for AI agents executing financial operations, addressing growing enterprise demand as "agentic commerce" rises.

Fragment’s core technology embeds “double-entry bookkeeping” logic into software to ensure transaction balances before funds move. This mechanism aims to fully record the source and destination of funds, thereby reducing the risk of significant financial losses caused by AI agent hallucinations, errors, or bypassing safety limits. Previously, Fragment had completed an $11 million seed round led by Stripe.

The company currently serves approximately 50 to 60 paying clients, including corporate credit card provider Ramp, financial software company BILL, and digital marketplace Whop. Although the valuation was not disclosed, notable investors and existing customers indicate that the market is beginning to prioritize security controls for AI-automated payments.

US-based AI company Anthropic announced on October 22, 2026, that it is donating its Agent Skills standard to the Agentic AI Foundation (AAIF), a project under the Linux Foundation. The foundation was co-founded last year by Anthropic, Block, and OpenAI, and previously received the Model Context Protocol (MCP) specification.

In an official statement, Anthropic said: "We’re committed to keeping Agent Skills open, community-driven, and vendor-neutral." This move means the standard will no longer be controlled by a single vendor but managed by a neutral body, aiming to reduce developer dependency on specific AI models and promote interoperability of agent skills across different platforms.

Prominent developer Simon Willison noted that this development aligns with his prediction from December last year, when he suggested Anthropic would likely donate the Skills standard to the AAIF. With both MCP and Agent Skills now housed in the same foundation, the underlying communication and capability extension mechanisms for AI agents are accelerating toward open standards.

Google announced Agent Compute Engine on October 22, 2026, a service designed to provide each AI agent with its own hardware-isolated micro virtual machine (microVM). The service currently powers Gemini internally and will open for preview in the coming months.

According to Google, these microVMs are secured by the company's Titanium chip, enabling hardware-level isolation. When an agent idles, its VM automatically pauses to free up resources for others. Shared memory between VMs significantly reduces the cost of running individual agents. Google claims that in test environments, this technology allows a single server to host ten times as many agents. In a test involving 200,000 agents, suspend and resume operations took less than 500 milliseconds.

This efficiency addresses the high overhead traditionally associated with VMs in agentic systems. As AI agents scale, resource isolation and scheduling have become critical infrastructure bottlenecks. Google also open-sourced Agent Substrate, a Kubernetes runtime supporting this architecture. On the governance side, Google added token tracking, model routing, and support for third-party identity providers like Ping Identity and SailPoint, extending these controls to Cloud Run, GKE, and managed endpoints.

Most teams can get an agent to complete a task once. Getting the same agent to complete that task on the 100th identical request, at a cost you can forecast, with a trace you can read when it fails, is a different engineering problem. Weights & Biases by CoreWeave blog author Shreyan Basu Ray attributed the core mechanism of this gap in a long-form piece published in July 2026 to one sentence: the cost driver is the loop resending accumulated context each iteration.

Specifically, a ten-step agent does not make ten small calls. It makes ten increasingly large ones. System prompts, tool schemas, conversation history, and every prior tool result are resent in each iteration. Data reported by Anthropic gives the shape: agent interactions use roughly 4x the tokens of chat interactions, and multi-agent systems roughly 15x. Agentic workloads routinely run input-to-output token ratios of 100:1 or worse, which means agentic cost is an input-token problem, not an output-token problem—intuitions built on chat pricing are off by an order of magnitude.

The article illustrates the mechanism with a concrete example of a customer service ticket triage agent: assume a 6,000-token system prompt with tool schemas, 8 steps, roughly 1,500 tokens of tool results and reasoning added per step, and 300 output tokens per step. A single run totals about 90,000 input tokens and 2,400 output tokens. At representative frontier pricing ($3 per million input tokens, $15 per million output tokens), that is approximately $0.31 per task. Compare that to a single classification call using the same system prompt, which costs about $0.02. The agent is roughly 14 times more expensive, and the entire difference comes from context resent across turns.

If the stable system prompt is cached (with a typical cache-read rate of one-tenth of base input), the same run drops to about $0.18, a reduction of about 42%. At a scale of 100,000 tasks per month, this single change is worth about $13,000. A systematic evaluation across three major providers on a multi-turn research benchmark found that prompt caching reduced API costs by 41% to 80%, improved time to first token by 13% to 31%, and savings scaled with prompt size, reaching nearly 90% at 50,000-token prompts.

But the article emphasizes a counterintuitive finding: naive full-context caching is not the optimal strategy. Caching only the stable system prompt was most consistent across both cost and latency, because full-context caching triggers cache writes for dynamic tool calls and results that will never be reused, adding overhead without corresponding reads. On one model, full-context caching produced a latency regression of about 9%, while system-prompt-only caching improved latency by about 31%. Practical rules are therefore very specific: keep the cacheable prefix stable, do not place timestamps, session identifiers, or user-specific strings at the beginning of the system prompt, because one differing token at the start invalidates the entire prefix.

Architecture choices impact cost far more than model selection. The article notes that the strongest published evidence for architecture-driven savings is model-tier orchestration: expensive models plan and delegate, cheap models execute. Anthropic reported in its multi-agent documentation that on the BrowseComp benchmark, an orchestrator model delegating to cheaper worker sub-agents reached 96% of the all-frontier configuration's performance at 46% of the cost—specifically, 86.8% accuracy at $18.53 per problem versus 90.8% accuracy at $40.56 per problem. The mechanism is simple: the orchestrator only sees plans and summaries, while workers absorb the bulk of the token-heavy reading and writing at the lower rate.

This saving holds under two conditions: workers must not genuinely need frontier reasoning for their subtasks, or you pay twice through retries; and repeat calls should land on the same worker so each keeps its own warm cache instead of re-paying full price for shared context.

A more subtle issue than cost is the honest measurement of success rates. τ-bench introduced pass^k—the fraction of k independent attempts at the same task that all succeed. The results are sobering: an agent resolving over 60% of retail tasks on a single attempt dropped to about 25% when required to succeed across eight attempts. A system that looks like a 60% performer is closer to one in four measured honestly. If your agent faces real users, the hundredth identical request matters more than the first.

The MAST Multi-Agent System Failure Taxonomy, based on analysis of over 1,600 annotated execution traces across seven frameworks, identified 14 failure modes. About 40% belong to specification and system design issues, about a third to inter-agent alignment failures, and the rest to verification failures. The authors explicitly note that improvements in base model capability are insufficient to cover the entire taxonomy—waiting for the next generation of models to fix reliability suggests a long wait, because the defects lie in the system design itself.

This cost structure directly rewrites priorities for inference infrastructure. In workloads with an input-to-output ratio of about 100:1, the majority of GPU time is spent processing inputs rather than generating outputs—the bottleneck is context, not generation. KV cache hit rate thus elevates from a tuning detail to a first-class operational metric, because a high hit rate directly eliminates the dominant cost.

The optimization order given by the article is: fix context first, then caching, then routing, and finally consider training, measuring after each step. Skipping intermediate steps to switch to a larger model usually reveals that you paid more to solve a problem that belonged in the context layer.

Gartner predicted in June 2025 that over 40% of agentic AI projects would be canceled by the end of 2027, citing escalating costs, unclear business value, and inadequate risk controls. Model capability is not on that list—the failures are economic and managerial, not capability-based.

Cost per call flatters agents. If a triage agent has a 70% success rate and retries once upon failure, the effective cost per successful task is much higher than the per-run number, because failed runs consume the full token budget before failing. Benchmarks or dashboards that ignore failed runs always make agents look cheaper than they are.

The same effect appears in published benchmark data: on the SWE-bench verified subset, multiple configurations reached identical 54.0% accuracy, but costs ranged from about $259 to about $1,790—same accuracy, roughly seven times the spend. Without a cost axis, leaderboards cannot tell you which to deploy.

There is a documented cliff when budgets tighten. In TestEvo-Bench, capping per-task spend at $1 dropped one agent's success rate on test generation from 70.6% to 44.2%, and another from 59.5% to 18.8%. Budget caps are a real control mechanism, but they are not free; different agents degrade in very different ways under budget pressure.

Token spend is just the visible part. Production budgets also bear tool and API call costs, retrieval and vector database infrastructure, trace storage and observability, evaluation overhead, human escalation handling—which is often the largest true unit cost when an agent routes to a person—and the engineering time to maintain tool contracts.

The tail matters more than the mean. Runs generated by agents that recursively spawn subagents or hit retry storms can cost orders of magnitude more than the median, and without per-run caps those runs are unbounded. Only by tagging cost per task in traces from day one and classifying by outcome can you calculate cost per successful task, cost per escalation, and the cost distribution.

The article simultaneously draws boundaries: when deterministic rules, SQL queries, classifiers, search indexes, or RPA scripts can reliably solve the task, agents are the wrong tool. Agents add latency, cost, nondeterminism, and operational surface area, requiring genuine uncertainty or adaptation problems to justify them. Choosing a deterministic alternative is not a step down; it is matching the tool to the problem.

There is an instructive divergence in the multi-agent field. Anthropic reported that its multi-agent research system outperformed single-agent configurations by 90.2% on internal evaluations, while also reporting a roughly 15x token multiplier and noting that token usage explained most of the performance variance. Cognition held a nearly opposite view in "Don't Build Multi-Agents": agents acting on partial context make conflicting decisions, so a single thread produces more coherent results. Both are right about different task shapes—the former is breadth-first research where subtasks are truly independent; the latter is coding, where each decision constrains the next.

The advice running through all choices is: choose the least autonomous architecture that satisfies the task, then add agentic flexibility only where the workflow genuinely needs runtime decisions. Architecture determines cost; task shape determines architecture.

The question in production environments is not whether an agent can complete a task once, but whether the workflow can fail safely, improve continuously under evaluation, and stay within latency and cost budgets. Every item in the current design consensus—from explicit termination conditions to typed tools to trajectory evaluation—exists because someone has already paid the tuition for not doing so. Compare against a deterministic baseline before scaling: if the agent cannot beat the baseline on measurable dimensions, that is also a useful result, and the cost learned now is far smaller than after launch.

New Enterprise Associates (NEA), an American venture capital firm, partner Philip Chopin and Hunter Worland opened their explanation for why they are leading Spiko’s $90 million Series B financing round for the Paris and London-based corporate cash management platform with a single sentence: too many businesses pay to lose money. More precisely, they pay management fees, performance fees, and transaction fees to hold cash that loses purchasing power every year on an inflation-adjusted basis.

The problem they identified is not that any specific bank offers low interest rates, but that the underlying infrastructure for corporate cash management remains stuck in the 20th century. Each layer is operated by a different institution, uses different ledgers, reconciles via overnight batch files, and exposes data to customers through outdated interfaces. The result is that businesses collectively lose billions of euros in yield annually, simply because the pipes beneath their cash have not been rebuilt.

Building on this diagnosis, Chopin and Worland define tokenization as the next foundational shift in financial infrastructure. Its mechanism involves migrating assets onto shared ledgers where ownership and balances exist on a single programmable ledger, enabling instant settlement, continuous yield accumulation, and software-driven workflows to replace batch processing and manual operations.

They further propose a forward-looking connection: this same structure serves as the natural foundation for AI agents that will increasingly run financial functions. On legacy bank and transfer agent infrastructure, automating finance is difficult because every action requires reconciling multiple siloed ledgers and waiting for overnight files; on a tokenized stack, AI agents can execute smart contracts, automatically sweep surplus cash into yield, post collateral, rebalance exposures within guardrails, and turn today’s static financial balances into always-on, self-optimizing systems.

Regarding why the approach starts in Europe, the authors cite three conditions: non-financial enterprises earn significantly lower yields than global peers, banks have underinvested in technology, and Spiko has already assembled a complete regulatory licensing stack. They simultaneously argue that the opportunity is fundamentally global, with the same architecture applicable to the United States, Latin America, and Asia.

Boundaries for readers to note: NEA is the lead investor in this round, and this text constitutes a public statement of its investment thesis. The cited figures of approximately $2.7 billion in assets under management and over 10,000 customers come from company self-reports as of September 2026, viewable on the public Dune dashboard, though NEA explicitly states in its disclaimer that certain information comes from third-party sources and has not been independently verified by them. The estimate of billions of euros in lost yield is provided by the authors without methodological detail. The assertion that AI agents run financial functions is a forward-looking judgment by the authors, not an achieved fact. Whether tokenized cash funds can evolve from supplementary tools for European corporate cash into mainstream infrastructure depends on the regulatory environment, institutional adoption speed, and operational frictions, which are not fully elaborated in this argument.

OmniParseBench, released by Datalab in 2026, is an open document parsing benchmark. It covers 2,937 pages of documents—including scanned forms, handwriting, arXiv mathematical formulas, and dense tables—using 16,288 pass/fail tests. It published item-level verdicts for 27 parsers, allowing anyone to download these verdict files and recalculate the leaderboard.

A developer did not simply cite the leaderboard numbers but downloaded the published verdict files and recalculated all items using the benchmark's own scoring function. The recalculation confirmed that all 108 numerical values on the leaderboard card are reproducible, but it also exposed three notable issues.

The first issue: the top two parsers are actually tied. Datalab accurate and Gemini 3.8 Flash both have a headline score rounded to 92.9, with a difference of only 0.024 percentage points. The developer used document-level bootstrap—sampling 2,343 documents with replacement 2,000 times and recalculating the headline each time—to estimate ranking uncertainty. The results showed that the 95% confidence interval for the gap between the two ranges from approximately -0.9 to +0.9, crossing zero. In each sample, each led roughly half the time. The honest interpretation is that they are indistinguishable.

Why use document-level rather than test-level bootstrap? Because tests on the same document tend to pass or fail together; independent sampling of individual tests would underestimate uncertainty. This choice is a key step in the audit methodology.

The second issue: rankings flip depending on document type. The developer recalculated headlines by slicing into 13 categories including handwriting, tiny fonts, rotated text, degraded scans, math, tables, forms, multi-column layouts, and scripts such as Greek, Chinese, Cyrillic, Armenian, and Lao. The overall leader, Datalab accurate, remained ahead in only 3 of these slices. Gemini 3.8 Flash led in handwriting, math, and forms; Claude Sonnet 5.5 led on rotated pages; and Mistral OCR led in Greek. The Armenian and Lao slices contained only 6 and 5 documents respectively, which the developer explicitly treated as hints rather than rankings.

The third issue: Datalab's own published figures are inconsistent with the files. The launch article and blog post listed Datalab accurate at 92.7 and ranked it behind Gemini, whereas the actual value in the files showed a difference of only 0.024 percentage points—a tie that rounds to the same figure. The developer verified the launch statements item by item; 13 of 17 checks held, while 4 did not (including Datalab's two ranking figures, the reproduction difference for Indra OCR, and the direction of the placebo test).

Another interesting finding was reported as a flag rather than proof. The OmniParseBench dataset card notes that Gemini 3.8 Flash and Claude Sonnet 5.5 participated in the creation and verification of some tests. The developer used a difference-in-differences test to ask: do these two star models perform abnormally well on the 102 tests corrected by vision models compared to other tests? The result showed that on the model_review slice, the pass rate drop for both models was about 25 percentage points less than for other parsers, while the human review placebo slice showed a smaller gap in the same direction, and the model_panel slice showed the opposite direction. The developer explicitly stated this is merely a flag and does not constitute proof.

Practical lessons: For any benchmark leaderboard, first confirm whether you can access the item-level verdict data; if so, use document-level resampling to estimate uncertainty rather than looking at a single headline number; recalculate rankings by slicing according to the document types you actually care about; verify whether vendor-reported figures match public files; and lock version numbers and dataset commits, as repositories change as new parsers are added.

The most common latency experience issue for voice agents is not that model generation is too slow, but that synthesis starts too late. A user asks a question, the language model is already generating a response token by token, but the text-to-speech module waits until the entire response is complete before it begins working. A single-sentence response might be generated in 600 milliseconds, but a full multi-paragraph response takes five seconds, and the integration logic waits for those five seconds. During these five seconds, the user hears nothing.

A technical solution released by the AWS SageMaker AI team on October 22, 2026, points out that the key to solving this problem is not running synthesis and generation in parallel, but changing the timing of synthesis startup: as soon as the language model produces a complete sentence, send that sentence immediately to the TTS model, without waiting for subsequent content. They deployed Kokoro-82M (with only 82 million parameters) on a SageMaker AI real-time endpoint, maintaining a persistent connection via a bidirectional WebSocket stream, sending text sentence by sentence and receiving audio sentence by sentence.

There is a potential misunderstanding here. The team explicitly clarifies that this is not concurrent synthesis—the loop of the bidirectional processor is strictly sequential: receive a message, synthesize its audio, send the audio, send a completion signal, then receive the next message. The audio for the second sentence does not begin generating while the first sentence is still being transmitted. What incremental sending actually achieves is that synthesis for the first sentence begins the instant it exists, rather than waiting for the full response. For multi-sentence responses, there is a genuine generation time gap between the first and last sentences; this gap represents the entire latency benefit.

Isolated measurements show that sending only the first sentence (rather than the full four-sentence text) reduced the time to first audio byte at the TTS transport layer from 329.3 milliseconds to 156.7 milliseconds, a relative improvement of 52.4%. In end-to-end comparisons, the bidirectional streaming mode was 2.4 times faster than batch mode (local container) and 3.5 times faster (deployed AgentCore runtime). The gap actually widens when real network hops are added, because the streaming mode absorbs network overhead within the overlap window of generation and synthesis, whereas every serial wait step in batch mode directly exposes network latency.

There is a second cost at the transport layer. If the frontend is already streaming text via Server-Sent Events (SSE), the team measured the cost of stuffing base64-encoded audio into SSE events: payload inflation is approximately 33%, and the SSE delimiter (empty line) does not align with internal MP3 frame boundaries—in practice, four logical MP3 frames arrived as six transmission blocks, two of which were truncated mid-frame. They compared MP3 at 48kbps and AAC 32kbps fMP4 encoding, confirming with Amazon Transcribe that decoding quality showed no difference.

For projects with an existing SSE pipeline, the base64 approach is a pragmatic choice, at the cost of 33% extra payload and a buffer reassembly step. For scenarios building a voice path from scratch or prioritizing latency, the team recommends directly reusing binary audio and JSON control frames on the same bidirectional session—the distinguishing rule is: if a data block cannot be parsed as a valid JSON object, it is audio. This rule works because although raw PCM audio may start with the byte 0x7b (left curly brace), the entire PCM frame will almost never coincidentally constitute a valid JSON object.

All data comes from self-tests conducted by the AWS team on their own SageMaker AI endpoints and Bedrock AgentCore runtime; independent third-party reproduction was not provided. The 82-million-parameter scale of Kokoro-82M is suitable for running on a single mid-tier GPU instance, but the article does not provide subjective quality assessments of synthesized speech or quality comparisons with other TTS models. The example container pins an older version of PyTorch with a known security advisory (PYSEC-2026-2286); the team notes that an upgrade is required before formal use.

This episode was published by technology media outlet Latent Space on October 22, 2026, featuring Erik Bernhardsson, co-founder and CEO of US-based AI infrastructure company Modal. The episode is titled 'Why AI Compute Comes in Rectangles' and focuses on elastic GPU supply, the rapid growth of sandbox businesses, and the economic logic behind the agent cloud.

According to the show notes, Bernhardsson explains the infrastructure and economics behind Modal's recent milestones, including crossing $1 billion in annualized run rate and raising a $750 million Series D. Key topics include why traditional fixed-duration GPU contracts ('rectangular' capacity) fail to match actual fluctuating compute demand; how Modal optimizes resource pools by aggregating demand from thousands of customers; and how the sandbox product evolved from being considered for shutdown to becoming a key business line accounting for roughly one-third of revenue. The discussion also covers alternatives to NVIDIA, the role of forward-deployed engineers (FDEs), and engineering culture.

According to a report by Semafor, cited by TechCrunch, Elon Musk has invested approximately $1 billion in Skydance, the newly formed media company. The news emerges weeks after Paramount completed its $110 billion acquisition of Warner Bros. Discovery to create the combined entity.

The report notes that Musk's investment coincided with equity contributions from other Skydance investors, including board member Laurene Powell Jobs. This capital provides a much-needed financial boost as the company seeks to cut costs by $6 billion over the next three years and lift its stock price.

Skydance is co-led by CEOs David Ellison and Ynon Kreiz, tasked with integrating streaming platforms Paramount+ and HBO Max alongside networks like CBS and CNN. Musk has close ties with Larry Ellison, whom he appointed to Tesla's board in 2018; Larry Ellison also invested $1 billion to help Musk purchase Twitter in 2022.

Skydance did not immediately respond to requests for comment. The specific terms, valuation basis, and final closing status remain unconfirmed by the issuer, presenting this strictly as a reported fact.

When training large models, networks face a rhythmic problem: thousands of GPUs simultaneously execute the same collective communication, over and over, for weeks. Bandwidth pressure is immense, but the pattern is predictable; engineers can design around this rhythm.

In a blog post published in August 2026, Min Jun of CoreWeave, an AI cloud infrastructure company, described another type of traffic: agents break tasks into steps, call retrievers, call tools, call another model, sometimes call another agent, and wait for each step to return before knowing what the next one is. This serial path is an agent chain, and its latency is the sum of every hop, not the worst single hop.

This distinction seems minor, yet it changes the logic of entire network design. When thousands of such chains run on the same hardware alongside training tasks and standard inference, traffic no longer resembles batch jobs; it becomes a ubiquitous, latency-sensitive, multi-hop conversation. Assumptions never tested by training begin to break.

The first consequence Min Jun explains is that oversubscription is not an option. When leaf switches have less uplink capacity to the spine than the nodes below them can generate, this gap remains invisible until two types of traffic compete for the same link. While training alone can tolerate some degree of oversubscription, a network mixing synchronized training bursts with thousands of small, sequential agent requests must simultaneously satisfy high aggregate bandwidth and low, predictable per-flow latency; oversubscribed links cannot achieve both. CoreWeave’s approach is to maintain a 1:1 non-blocking ratio at every layer.

The second consequence involves how the network scales. Traditional two-tier leaf-spine topologies eventually exhaust switch radix limits; the solution is adding a third tier, but this increases hop count—which is precisely what agent chains cannot afford. CoreWeave adopts a multi-plane topology: distributing each GPU’s SuperNIC connections across two or more independent network planes, trading depth for width. Min Jun states this allows a flat, two-tier design to scale to 128,000 GPUs, 64 times the coverage of a single-plane network, without requiring redesign.

The third consequence is that load balancing must move from software to hardware. Traditionally handled by NCCL at the software layer to disperse collective communication traffic across available links, this approach carries two costs: competing with host CPUs for cycles, and only reacting to congestion after the fact rather than proactively routing around it as conditions change. CoreWeave uses NVIDIA ConnectX-9 SuperNICs to offload this task to hardware, allowing dedicated hardware paths to respond to real-time congestion telemetry at line rate. In its NVIDIA Vera Rubin NVL72 deployment, each GPU’s SuperNIC runs at up to 1,600 Gb/s; switches coordinate directly with SuperNICs, dispersing traffic across every available path, plane, and rail using real-time telemetry rather than static hashing.

The fourth consequence is that resilience cannot be centralized. A training job stopping for a few minutes is expensive, but it can recover from checkpoints. An agent stuck halfway through a ten-step chain, multiplied across thousands of concurrent chains, has no equivalent recovery point. Min Jun describes building resilience across five layers: the topology layer shifts traffic off degraded paths using multi-plane and multi-rail designs; the silicon layer uses BlueField-3 DPUs to place networking, isolation, and policy enforcement on dedicated hardware, freeing host CPU cycles; the software layer uses CoreWeave Kubernetes Service (CKS) to completely remove the hypervisor, ensuring virtualization faults do not become tenant issues; the operations layer uses Mission Control to automatically replace nodes once they cross health thresholds; and the geographic layer links multiple data centers into a single backbone, ensuring cross-site workloads do not depend on any single facility.

The fifth consequence is that observability must reach every hop. If the slowest hop is hidden, all previous design efforts are futile. CoreWeave uses NVIDIA network observability tools to trace flow-level performance from GPU to SuperNIC, transforming the vague symptom of “this agent chain feels slow” into specific links or queues that engineers can address.

Min Jun concludes by writing: "Training got the industry to build fast networks; agentic AI will determine which of those networks were built to last."

It should be noted that this is a firsthand description of CoreWeave’s own architecture. Claims regarding the scale of its operations, including having one of the largest InfiniBand and RoCE networks and one of the largest DPU deployments, are self-reported and not independently verified. The figure of 128,000 GPUs represents the design upper limit of the multi-plane topology; the article does not specify whether this is a deployed or planned configuration. All performance figures come from vendor self-testing, not third-party benchmarks.

US-based AI cloud infrastructure company CoreWeave announced on July 28, 2026, via its official blog that it achieved leading results in the inaugural MLPerf 0.7 Endpoints benchmark by MLCommons. Running the frontier DeepSeek-R1 reasoning model on a NVIDIA GB200 NVL72 system with 68 NVIDIA Blackwell GPUs, CoreWeave reported a sustained system throughput of 441,740 output tokens per second at 16,384 concurrent requests, and up to 6,496 tokens per second per GPU.

MLPerf Endpoints is MLCommons' newest benchmark suite, designed to measure how an inference service behaves under realistic, concurrent, online load, reporting the full throughput-versus-latency curve rather than a single data point. The test includes DeepSeek-R1, a 671-billion-parameter mixture-of-experts reasoning model that emits thousands of tokens per response, making it the hardest serving workload in the suite.

CoreWeave emphasized that its results were validated on standard production clusters using the same network fabric, scheduler, storage, and Mission Control orchestration that customers use, rather than an isolated environment tuned for the test. The company noted that throughput per GPU is the metric most directly reflecting the economics of real inference services, as it maps to tokens served per accelerator and cost per GPU.

It should be noted that these performance figures are self-reported by CoreWeave. While the submission process was verified by the MLCommons Association, independent third-party reproduction has not been publicly reported. Additionally, the 6,496 tokens per second per GPU figure is derived by dividing system TPS by the GPU count (68) and is not a direct MLPerf reported metric.

The hardest problem to diagnose in production inference services is not downtime, but 'not down but constantly slowing down.' In a blog post published in June 2026, CoreWeave authors Tara Madhyastha and Nisha Nadkarni illustrate this dilemma with a hypothetical scenario: a medical question-answering service runs for three weeks without failure alerts, error spikes, or dashboard warnings—all indicators are green—but user complaints begin arriving: responses are slower, with mid-answer stuttering. Pulling metrics reveals that p99 time-to-first-token (TTFT) latency climbed from 180ms to 240ms, and peak-hour availability slipped from 99.9% to 99.1%, neither crossing alert thresholds. Users had experienced nearly two weeks of quality degradation, yet the system was never classified as 'faulty'.

The authors attribute this silent drift to three structural causes that compound as load grows. First is infrastructure-layer resource contention: general-purpose cloud infrastructure is designed for flexible, heterogeneous workloads, whereas inference is continuous, latency-sensitive, and highly sensitive to resource competition. At small scale, requests rarely compete; at production scale, GPU contention becomes constant, and scheduling overhead scales with request volume. Latency negligible at 50 requests per second becomes significant at 5,000. Noise from neighbors on shared network paths appears first in tail latency; p50 looks fine, but p99 tells a different story.

Second is model-serving configuration drift. The authors use an analogy to explain batching: static batching is like a tour bus, waiting until full before departing, forcing fast requests to wait for the slowest one; continuous batching is like a subway, where passengers board and alight at every stop, keeping the GPU fully loaded. However, the subway has its own issues: a long incoming prompt (prefill) can block the platform for all requests already in transit (decode), causing inter-token latency jitter. KV cache pressure is another path: as context windows lengthen and sessions multiply, the engine begins evicting or preempting in-flight requests to free space. Recomputation overhead does not show up in error rates but does show up in p99.

Third is the mismatch between traffic patterns and autoscaling. Autoscaling systems respond to observed demand with inherent lag: a new pod takes roughly 90 seconds to come online, and a new node requires several minutes. When burst traffic arrives, autoscaling cannot keep up, requests queue, latency climbs, and if sustained, requests begin to timeout. The authors also point out a more subtle version: unequal request shapes. If infrastructure is sized for average complexity, when the heaviest requests cluster together, capacity saturates faster than the autoscaling policy expects.

For these three causes, the article provides three diagnostic signals: climbing p99 with stable p50 points to infrastructure variability or configuration drift rather than load issues; rising TTFT under stable load points to GPU contention or batching configuration; and a gap between goodput (requests meeting latency SLAs) and throughput (total processed requests) indicates the system is 'running' but some users are actually experiencing failures—a system with 95% throughput may still be timing out one-fifth of users, while monitoring never records this failure.

The authors' core conclusion is that teams must treat reliability as an architectural property rather than a monitoring problem, choosing explicit GPU allocation, traffic-aware autoscaling, workload-validated SLAs, and predictable networking. It is worth noting that CoreWeave itself sells dedicated inference infrastructure, so this conclusion aligns with its commercial positioning; all numbers in the text are hypothetical examples, not measured production data. Well-scheduled general-purpose cloud environments, combined with topology-aware scheduling and pre-allocation strategies, can also mitigate these issues to a considerable extent.