Reading

When training large models, networks face a rhythmic problem: thousands of GPUs simultaneously execute the same collective communication, over and over, for weeks. Bandwidth pressure is immense, but the pattern is predictable; engineers can design around this rhythm.

In a blog post published in August 2026, Min Jun of CoreWeave, an AI cloud infrastructure company, described another type of traffic: agents break tasks into steps, call retrievers, call tools, call another model, sometimes call another agent, and wait for each step to return before knowing what the next one is. This serial path is an agent chain, and its latency is the sum of every hop, not the worst single hop.

This distinction seems minor, yet it changes the logic of entire network design. When thousands of such chains run on the same hardware alongside training tasks and standard inference, traffic no longer resembles batch jobs; it becomes a ubiquitous, latency-sensitive, multi-hop conversation. Assumptions never tested by training begin to break.

The first consequence Min Jun explains is that oversubscription is not an option. When leaf switches have less uplink capacity to the spine than the nodes below them can generate, this gap remains invisible until two types of traffic compete for the same link. While training alone can tolerate some degree of oversubscription, a network mixing synchronized training bursts with thousands of small, sequential agent requests must simultaneously satisfy high aggregate bandwidth and low, predictable per-flow latency; oversubscribed links cannot achieve both. CoreWeave’s approach is to maintain a 1:1 non-blocking ratio at every layer.

The second consequence involves how the network scales. Traditional two-tier leaf-spine topologies eventually exhaust switch radix limits; the solution is adding a third tier, but this increases hop count—which is precisely what agent chains cannot afford. CoreWeave adopts a multi-plane topology: distributing each GPU’s SuperNIC connections across two or more independent network planes, trading depth for width. Min Jun states this allows a flat, two-tier design to scale to 128,000 GPUs, 64 times the coverage of a single-plane network, without requiring redesign.

The third consequence is that load balancing must move from software to hardware. Traditionally handled by NCCL at the software layer to disperse collective communication traffic across available links, this approach carries two costs: competing with host CPUs for cycles, and only reacting to congestion after the fact rather than proactively routing around it as conditions change. CoreWeave uses NVIDIA ConnectX-9 SuperNICs to offload this task to hardware, allowing dedicated hardware paths to respond to real-time congestion telemetry at line rate. In its NVIDIA Vera Rubin NVL72 deployment, each GPU’s SuperNIC runs at up to 1,600 Gb/s; switches coordinate directly with SuperNICs, dispersing traffic across every available path, plane, and rail using real-time telemetry rather than static hashing.

The fourth consequence is that resilience cannot be centralized. A training job stopping for a few minutes is expensive, but it can recover from checkpoints. An agent stuck halfway through a ten-step chain, multiplied across thousands of concurrent chains, has no equivalent recovery point. Min Jun describes building resilience across five layers: the topology layer shifts traffic off degraded paths using multi-plane and multi-rail designs; the silicon layer uses BlueField-3 DPUs to place networking, isolation, and policy enforcement on dedicated hardware, freeing host CPU cycles; the software layer uses CoreWeave Kubernetes Service (CKS) to completely remove the hypervisor, ensuring virtualization faults do not become tenant issues; the operations layer uses Mission Control to automatically replace nodes once they cross health thresholds; and the geographic layer links multiple data centers into a single backbone, ensuring cross-site workloads do not depend on any single facility.

The fifth consequence is that observability must reach every hop. If the slowest hop is hidden, all previous design efforts are futile. CoreWeave uses NVIDIA network observability tools to trace flow-level performance from GPU to SuperNIC, transforming the vague symptom of “this agent chain feels slow” into specific links or queues that engineers can address.

Min Jun concludes by writing: "Training got the industry to build fast networks; agentic AI will determine which of those networks were built to last."

It should be noted that this is a firsthand description of CoreWeave’s own architecture. Claims regarding the scale of its operations, including having one of the largest InfiniBand and RoCE networks and one of the largest DPU deployments, are self-reported and not independently verified. The figure of 128,000 GPUs represents the design upper limit of the multi-plane topology; the article does not specify whether this is a deployed or planned configuration. All performance figures come from vendor self-testing, not third-party benchmarks.

US-based AI cloud infrastructure company CoreWeave announced on July 28, 2026, via its official blog that it achieved leading results in the inaugural MLPerf 0.7 Endpoints benchmark by MLCommons. Running the frontier DeepSeek-R1 reasoning model on a NVIDIA GB200 NVL72 system with 68 NVIDIA Blackwell GPUs, CoreWeave reported a sustained system throughput of 441,740 output tokens per second at 16,384 concurrent requests, and up to 6,496 tokens per second per GPU.

MLPerf Endpoints is MLCommons' newest benchmark suite, designed to measure how an inference service behaves under realistic, concurrent, online load, reporting the full throughput-versus-latency curve rather than a single data point. The test includes DeepSeek-R1, a 671-billion-parameter mixture-of-experts reasoning model that emits thousands of tokens per response, making it the hardest serving workload in the suite.

CoreWeave emphasized that its results were validated on standard production clusters using the same network fabric, scheduler, storage, and Mission Control orchestration that customers use, rather than an isolated environment tuned for the test. The company noted that throughput per GPU is the metric most directly reflecting the economics of real inference services, as it maps to tokens served per accelerator and cost per GPU.

It should be noted that these performance figures are self-reported by CoreWeave. While the submission process was verified by the MLCommons Association, independent third-party reproduction has not been publicly reported. Additionally, the 6,496 tokens per second per GPU figure is derived by dividing system TPS by the GPU count (68) and is not a direct MLPerf reported metric.

The hardest problem to diagnose in production inference services is not downtime, but 'not down but constantly slowing down.' In a blog post published in June 2026, CoreWeave authors Tara Madhyastha and Nisha Nadkarni illustrate this dilemma with a hypothetical scenario: a medical question-answering service runs for three weeks without failure alerts, error spikes, or dashboard warnings—all indicators are green—but user complaints begin arriving: responses are slower, with mid-answer stuttering. Pulling metrics reveals that p99 time-to-first-token (TTFT) latency climbed from 180ms to 240ms, and peak-hour availability slipped from 99.9% to 99.1%, neither crossing alert thresholds. Users had experienced nearly two weeks of quality degradation, yet the system was never classified as 'faulty'.

The authors attribute this silent drift to three structural causes that compound as load grows. First is infrastructure-layer resource contention: general-purpose cloud infrastructure is designed for flexible, heterogeneous workloads, whereas inference is continuous, latency-sensitive, and highly sensitive to resource competition. At small scale, requests rarely compete; at production scale, GPU contention becomes constant, and scheduling overhead scales with request volume. Latency negligible at 50 requests per second becomes significant at 5,000. Noise from neighbors on shared network paths appears first in tail latency; p50 looks fine, but p99 tells a different story.

Second is model-serving configuration drift. The authors use an analogy to explain batching: static batching is like a tour bus, waiting until full before departing, forcing fast requests to wait for the slowest one; continuous batching is like a subway, where passengers board and alight at every stop, keeping the GPU fully loaded. However, the subway has its own issues: a long incoming prompt (prefill) can block the platform for all requests already in transit (decode), causing inter-token latency jitter. KV cache pressure is another path: as context windows lengthen and sessions multiply, the engine begins evicting or preempting in-flight requests to free space. Recomputation overhead does not show up in error rates but does show up in p99.

Third is the mismatch between traffic patterns and autoscaling. Autoscaling systems respond to observed demand with inherent lag: a new pod takes roughly 90 seconds to come online, and a new node requires several minutes. When burst traffic arrives, autoscaling cannot keep up, requests queue, latency climbs, and if sustained, requests begin to timeout. The authors also point out a more subtle version: unequal request shapes. If infrastructure is sized for average complexity, when the heaviest requests cluster together, capacity saturates faster than the autoscaling policy expects.

For these three causes, the article provides three diagnostic signals: climbing p99 with stable p50 points to infrastructure variability or configuration drift rather than load issues; rising TTFT under stable load points to GPU contention or batching configuration; and a gap between goodput (requests meeting latency SLAs) and throughput (total processed requests) indicates the system is 'running' but some users are actually experiencing failures—a system with 95% throughput may still be timing out one-fifth of users, while monitoring never records this failure.

The authors' core conclusion is that teams must treat reliability as an architectural property rather than a monitoring problem, choosing explicit GPU allocation, traffic-aware autoscaling, workload-validated SLAs, and predictable networking. It is worth noting that CoreWeave itself sells dedicated inference infrastructure, so this conclusion aligns with its commercial positioning; all numbers in the text are hypothetical examples, not measured production data. Well-scheduled general-purpose cloud environments, combined with topology-aware scheduling and pre-allocation strategies, can also mitigate these issues to a considerable extent.

Canadian artificial intelligence company Cohere announced on October 22, 2026, that its enterprise model inference service, Model Vault, now supports Confidential Computing and has entered public beta. The feature leverages Microsoft Azure infrastructure and NVIDIA H100 GPU confidential computing capabilities to provide hardware-level protection for sensitive workloads in sectors such as finance, healthcare, and law.

While traditional encryption protects data at rest and in transit, confidential computing targets data in use. In Model Vault's new architecture, inference runs within a hardware-isolated environment spanning both CPU and GPU. NVIDIA's confidential computing technology ensures that GPU memory is protected and traffic between the CPU and GPU is encrypted while the model performs inference. This means that even cloud providers or Cohere's infrastructure operators cannot access plaintext data.

Beyond hardware isolation, Cohere has introduced a remote attestation mechanism. Customers can cryptographically verify the processing environment before sending sensitive data, checking whether the hardware, firmware, and software configuration meet approved security policies. If the environment fails verification, the client refuses to send the request. This shifts the trust model from "trusting vendor promises" to "verifying technical controls yourself."

Additionally, Model Vault implements end-to-end encryption for application payloads. Request contents remain encrypted until they enter the confidential environment, where they are decrypted only within that trusted boundary, with responses similarly protected on the return path. Phil Blunsom, CTO at Cohere, stated that this allows enterprises to avoid choosing between the simplicity of managed AI and control over their most sensitive information.

The August survey by the Charter research organization covered over 5,000 US respondents who reported using AI at work. Results show that approximately one-third of executives trust AI's judgment more than their own, while this proportion is only about half as high among ordinary employees. On the question of 'who truly benefits from AI,' 55% of executives stated that AI improved both their work and personal lives, whereas only about half of the employees gave the same response.

The article by t3n.de points out a structural contradiction: companies hire executives for their experience and knowledge, not to hand over decisions to AI. When those most responsible for providing judgment begin to offload it, the issue is no longer just personal preference but the erosion of organizational capability.

The article introduces a specific term to describe this phenomenon: 'Cognitive Offloading,' which refers to the behavior of transferring cognitive work to AI systems to reduce one's own cognitive load. Having AI summarize a long article rather than reading it oneself, or having AI analyze resumes or financial data rather than reviewing them personally—these practices indeed save time and mental energy. However, the cost of saving cognitive load may be the cognitive capacity itself.

An experiment by the MIT Media Lab also showed that subjects who delegated fact-checking tasks to AI systems for a month had reduced ability to detect misinformation when the AI tool was no longer available. Anku Rani, a doctoral student and one of the study's lead authors, warned in an MIT publication that users often forget that AI models make statistical predictions, not true judgments.

The Center for Universal Education, part of the Brookings Institution, observed a similar pattern among adolescents: when students hand over their thinking processes to AI, the educational process may be undermined. 65% of surveyed students expressed concern about the long-term decline of their cognitive abilities. One student's quote was: 'Simply put, you don't need to use your brain.'

These pieces of evidence collectively point to a 'Deskilling' mechanism, analogous to how calculators weakened mental arithmetic skills in historical precedents. The difference is that calculators handle deterministic calculations, while AI handles judgmental tasks—the latter being precisely the reason executives are hired. When statistical predictions are used as judgment, hallucinations, misleading inferences, and biases enter the decision chain, affecting scope from individual executives to the entire organization.

It must be clarified that the MIT experiment measured a specific ability—fact-checking—rather than general management judgment. The survey data reflects self-reported trust levels, not direct measurements of decision quality. There is an extrapolation between 'decline in discrimination ability after relying on AI for fact-checking for one month' and 'degeneration of judgment after executives delegate data analysis to AI,' involving differences in task complexity and time span. However, consistent directional evidence is sufficient to indicate that cognitive offloading is not a zero-cost efficiency tool; it may be eroding core capabilities in ways least noticeable to the user.

At the 'OpenAI DevDay Exchange: Seoul' event held on October 22, 2026, Albert Yip, Head of Technical Success at OpenAI Asia-Pacific, announced a new mechanism allowing paid ChatGPT users to share their included AI usage limits with third-party applications. This feature is called 'Sign in with ChatGPT'.

Previously, developers integrating OpenAI models into their apps had to bear the API costs incurred by each user call. Under the new system, when users log in via their ChatGPT account in a supported app and agree to share usage, the AI consumption within that app is deducted directly from their existing ChatGPT Plus or Pro subscription quota. This means developers do not need to pay extra inference costs for trial users, while users can make fuller use of their purchased subscription services.

OpenAI stated that the first batch of partners supporting this feature includes 16 companies such as AI coding tool Devin, collaboration platform Notion, and web development platform Vercel. It is important to note that usage in external apps is not unlimited but shares the same limit as the main subscription. Additionally, account login and usage sharing are two separate authorization steps to protect user privacy and control. Currently, the feature is primarily targeted at open-source tools and personal projects; commercial applications must apply and undergo review to integrate.

US-based AI company Perplexity has updated its macOS agent system, "Perplexity Computer," to support local Large Language Model (LLM) execution. Previously dependent on cloud models, the app now allows users to perform AI tasks locally without an internet connection. Initially supported models include Qwen 3.6 and PPLX Pro, a Qwen variant post-trained by Perplexity itself.

Local operation imposes strict hardware requirements: it is limited to Apple Silicon Macs with at least 36 GB of RAM recommended. Perplexity suggests newer devices, such as the M5 Pro Mac mini with 48 GB memory, which costs over €2,659. Access to this feature is not free; it is restricted to customers with Pro, Max, or Enterprise subscriptions.

Perplexity states that complex multi-step tasks, such as analyzing private financial portfolios or working with local files and emails, can be performed entirely locally, ensuring no data leaves the machine. For cases where local performance is insufficient, the app offers a "Hybrid Compute" option, allowing a local orchestrator to activate one of over 15 frontier cloud models. Independent tests are still needed to verify the seamless switching between local and cloud modes and the overall optimization of these local models.

When traffic peaks arrive, text-to-speech (TTS) requests begin queuing, and latency breaches the service level objective (SLO)—even though auto-scaling policies were supposed to prevent this. The problem lies in the fact that the thresholds these policies rely on are often guessed based on experience. If guessed too conservatively, GPU instances sit idle, costing money on every invoice without triggering any alerts. If guessed too aggressively, each instance carries more concurrent requests than its latency promise allows. Everything looks normal in the console until a real traffic spike pushes queueing time past the SLO, exposing the failure at the moment most callers are listening.

Why can't TTS use the same scaling metrics as large language models? Most public guidelines target short synchronous request/response calls or LLM inference, where the natural metric is tokens generated per second. However, non-autoregressive models like Kokoro-82M produce full audio from a single forward pass, lacking the token-by-token generation loop needed to attach such metrics. What determines whether a TTS instance is overloaded is simpler: how many requests are currently queued and executing. Therefore, this solution uses SageMaker AI's native ConcurrentRequestsPerModel metric instead of custom throughput measures.

The core method is called the four-number model: operators select two parameters—the latency SLO (3,000 ms p95 time-to-first-audio-byte in this case) and an overflow ratio (1.25x, allowing traffic to grow by 25% within one scaling cycle without breaching the SLO)—then perform two measurements on the live endpoint. First is Q_max: using a closed-loop load ladder, increase concurrency step-by-step until p95 latency breaches the SLO, taking the last concurrency value that met the SLO. For Kokoro-82M on ml.g5.xlarge, Q_max was measured at 50. Second is T_total: force a scale-out event, timing each stage from capacity request to healthy traffic recovery. In this case, pure provisioning took 260.2 seconds; adding the upper bound of policy detection lag (60 seconds), the planned T_total is 320.2 seconds.

With Q_max and T_total, scaling thresholds and cooldowns are derived arithmetically: Scale-out threshold C_scale_max = (1−h) × Q_max = 0.75 × 50 = 37.5; Scale-in threshold C_scale_min = (1−2h) × Q_max = 0.50 × 50 = 25. The scale-out cooldown is short (30 seconds) because the cost of adding one instance is low. The scale-in cooldown is long (approximately 3 times T_total, i.e., 960 seconds) because if traffic rises again after removing an instance, re-scaling requires the full 320 seconds. The container’s own admission control hard-limits queue depth at Q_max=50, returning a 503 rejection for excess requests rather than letting them queue to inevitable timeout.

The key insight of this method extends beyond TTS. The blog notes that for latency-sensitive, concurrency-bound real-time inference endpoints on SageMaker AI—such as an image embedding endpoint keeping connections open during GPU forward passes—the bottleneck is also concurrent in-flight requests, not throughput metrics borrowed from other workload shapes. The derivation logic and five-step toolchain (drift, qmax, ttotal, plan, validate) can be migrated directly. However, specific values differ by model and instance type: Q_max=50 and T_total=320 seconds apply only to Kokoro-82M on ml.g5.xlarge; different models or machine types require new measurements.

Ben Carlson wrote on his blog "A Wealth of Common Sense" that a reader, aged 38 with all assets in broad-based index funds, asked him: "I recently heard you bought bonds for the first time in your career; how did you decide which type to buy?" His answer was not "because interest rates are going to fall," but rather a complete line of reasoning extending from duration sensitivity to starting yields to life-stage risk budgets.

Duration is the core concept of this article. Carlson explains that duration is essentially interest rate sensitivity—a bond with a duration of 5 years will see its price drop by approximately 5% for every 1 percentage point rise in interest rates; conversely, if interest rates fall by 1 percentage point, the price will rise by approximately 5%. He gives an example that the U.S. long-term Treasury ETF (TLT) currently has a duration of about 15 years, meaning a 1 percentage point rise in interest rates would result in a price loss of approximately 15%. "I don't like guessing interest rate trends," he wrote, "I prefer holding intermediate-to-short term durations of 5 to 6 years or shorter."

Behind this choice lies a principle he repeatedly emphasizes: only bear volatility when paid a risk premium. The volatility of long-term Treasuries is too large, so holding them is essentially speculating on the direction of interest rates rather than collecting fixed income. Intermediate-to-short term bonds provide a relatively certain source of return—"collecting interest"—while keeping the price impact caused by interest rate changes within an acceptable range.

So why now? The direct reason Carlson gives is the starting yield. "Bond returns tend to track starting bond yields," he wrote, "Expected returns for bonds are higher than they have been at any time since Apple released its first iPhone." For the past twenty years—from the ultra-low interest rates after the U.S. financial crisis to further suppression during the pandemic—fixed income provided almost no decent income. Now yields have returned to their highest levels since 2007, changing the math.

But he simultaneously incorporates personal life stage into the decision. "I am also at a different stage in my life; I am middle-aged, and my goals are different." He bought some tax-efficient bonds for his brokerage account, which might be used for vacations, home renovations, ammunition for the next stock market downturn, or simply to "collect 5.5% interest and be happy." He still holds significant equity risk in both his retirement accounts and his brokerage account.

Finally, he clearly defines what he is NOT doing: "This is not predicting a stock market crash; the bull market could certainly continue. This is not predicting an interest rate collapse; if inflation remains sticky, the U.S. government continues spending, and AI capital expenditures soar, bond yields could absolutely continue to rise." He admits that if interest rates rise, his bond prices will fall in the short term, but believes that the current higher yields offer a "margin of safety" that hasn't existed for some time—and if interest rates do continue to rise, it means that once the dust settles, expected returns will be even higher.

"Or maybe I'm just getting old and boring," he wrote, "but either way, bond yields look attractive for the first time in a long time."

The exponential growth of AI compute rests on a physical prerequisite: someone must manufacture the chips. Intel Chief Technology Officer Pushkar Ranade provided a specific figure during an episode of the Dwarkesh Patel podcast aired on October 22: even with unlimited capital, the global stock of existing chipmaking tools can at most double leading-edge wafer production. Meanwhile, the AI industry’s compute growth rate over the past decade has been threefold per year.

What does this gap between these two figures imply? Ranade’s answer is that output cannot increase unless the entire supply chain—from equipment suppliers to their suppliers and sub-suppliers—produces more. He specifically highlighted that ASML produces fewer than 100 EUV lithography machines annually, with no plans to significantly exceed this number by the end of the current decade. These machines must be allocated among Intel, TSMC, and all memory manufacturers.

The deeper constraints extend beyond EUV. Ranade pointed out that once a process flow is locked in—for example, the now-mature 18A node—specific tools are required, and one cannot simply switch to equipment from another supplier. Even if multiple suppliers exist for certain tools, process lock-in means reliance on a single source remains effectively true. Equipment delivery cycles are at least 18 months: placing massive orders today means receiving equipment no earlier than 2028; if new fab shells need to be built as well, that pushes delivery to the end of the decade.

This does not mean capacity growth is impossible. Ranade distinguished between two dimensions: marginal output can be increased on a 12-month scale (by optimizing utilization of existing equipment), but total volume growth through the end of the decade will be extremely limited. Intel’s own experience corroborates this: moving from 18A to 14A allows reuse of approximately 80% of the toolset, but denser geometries require new equipment.

So what drives continued AI compute growth? In the interview, Ranade and Chris Auth, who leads development of all Intel’s mass-production processes, offered another path: making each chip on every wafer more powerful. Auth described the evolution from planar transistors to FinFETs to gate-all-around architectures, and the next step of CFET (Complementary Field-Effect Transistors, which stack n-type and p-type transistors). However, Auth admitted their visibility into technology roadmaps extends only about four years; predicting specific forms beyond a decade is difficult.

What does this constraint mean for AI readers? It indicates that AI compute’s exponential growth is not automatic. Hardware capacity has a hard upper limit determined by the equipment supply chain. Breaking through it requires either continuous improvement in per-chip performance (new architectures, new materials) or synchronized expansion across the entire supply chain (which itself is constrained by capital, time, and physical conditions). The 2X cap cited by Intel’s CTO is the most direct quantitative expression of this hard constraint.

Travis Knight, CEO of stop-motion animation studio Laika and director of the new film "Wildwood," drew a clear distinction between pencils, paintbrushes, 3D printers, and workstations, and generative AI during an interview with The Verge: the former have meaning and value when placed in the hands of an artist, while the latter represent 'technology doing the artist's work.' He stated his firm opposition to the latter, explaining that 'when technology consumes the material that artists truly create and then regurgitates it in a different way, there is no soul, no spirit, no humanity behind it—that is theft.' He further elaborated that only those who have experienced love, loss, and heartbreak can tell stories about these experiences, noting that computers have never had such experiences. Knight simultaneously acknowledged that AI may bring great benefits in the fields of medicine and science, making it clear that he is not opposed to all AI applications, but rather concerned about how it is being used.

In 2016, the Google DeepMind team published the paper "Mastering the game of Go with deep neural networks and tree search" in Nature magazine, introducing the AlphaGo program. This research addressed the classic AI challenge of Go by proposing a new method combining "value networks" and "policy networks." These deep neural networks were trained through supervised learning from human expert games and reinforcement learning from self-play.

The core innovation of the paper lies in combining Monte Carlo Tree Search (MCTS) with neural networks. The policy network is used to select move positions, while the value network evaluates board states. Without lookahead search, the neural networks alone achieved the level of state-of-the-art simulation-based random self-play programs at the time. With the introduction of the new search algorithm, AlphaGo achieved a 99.8% win rate against other Go programs and defeated European Go champion Fan Hui 5-0. This was the first time a computer program defeated a professional human player in a full-sized game of Go, an achievement previously thought to be at least ten years away.

For modern readers seeking to understand the origins of current large model reasoning capabilities, the architectural ideas in this paper are relevant. For example, consider a complex decision system that no longer relies solely on rule engines or simple statistical probabilities but, like AlphaGo, uses neural networks to "intuitively" filter out a few most promising action options (policy network), then performs deeper simulation evaluations on these options to judge the final outcome (value network). This hierarchical processing mechanism of "fast filtering + deep evaluation" is one of the key approaches to solving high-dimensional space search problems.

Recommended reading: "Mastering the game of Go with deep neural networks and tree search" (2016). For researchers and engineers interested in how deep learning applies to complex games and how prior knowledge of neural networks guides search algorithms, this is a foundational document. It is suggested to start with the abstract and Figure 1 (neural network training flow), focusing on the role of policy and value networks in node expansion within Monte Carlo Tree Search.