ReadingResearchRadarInvestment framework
Sign in / Sign up
AllTechnologyProductsInvestingPublic marketsOther

Reading

2026-11-1734 posts

Refresh

Microchip Announces ATFCD52 Tactile Sensing Chip for Robotic Slip Detection at Up to 800Hz

Quick take
2026-11-17 21:30 GMT+8

US semiconductor company Microchip Technology announced the ATFCD52, a high-speed capacitive digitizer integrated circuit, in November 2026. The chip is designed to enhance tactile sensing in robotic hands, capable of collecting data at frame rates up to 800Hz and detecting capacitance changes smaller than 1 femtofarad.

By processing this high-frequency tactile data, systems can calculate normal and shear forces to determine whether an object is being held securely or beginning to slip. This allows robots to adjust their grip force in real time before dropping objects, improving operational safety and reliability.

The ATFCD52 supports up to 52 channels and includes 20 parallel analog front-ends to simultaneously collect data from all receive lines. Beyond robotics, Microchip plans to apply this technology to AR/VR accessories, gaming gloves, and wearable sensors.

Sources:EE Times Japan

Research

AMD Releases ROCm HRX Loom SDK v0.2 with Expanded C++ Support and Optimized GPU Execution

Quick take
2026-11-17 09:37 GMT+8

U.S. semiconductor company AMD released version 0.2 of the ROCm HRX Loom SDK on November 16, 2026. Loom is part of the ROCm HIP Runtime Extended (HRX) system, serving as the compiler, runtime, and infrastructure for supporting AI agent workflows. This update builds upon v0.1 by expanding C++ program support, packaged device execution capabilities, and native AMD GPU launch and replay support.

According to the official release notes, Loom 0.2 introduces complete-program C++ authoring, runtime-provided managed objects, and stateful finite services. It extends the shared target compiler across x86, AIE2P, AMDGPU, SPIR-V, and WebAssembly platforms. Additionally, the new version removes fixed and superlinear costs from JIT compilation, graph construction and updates, GPU launch and replay, pipeline planning, and long-context serving, thereby improving overall efficiency.

Regarding the hardware abstraction layer, Loom tools, examples, serving, sanitizers, and hardware qualification now use one libamdf-backed AMD HAL through the existing --device=amdgpu spelling. This native path supports RDNA and CDNA architecture execution, including shared CPU/GPU/NPU memory contracts, explicit compute-resource selections, cooperative execution, virtual mappings, and reusable command buffers. The former ROCr-backed driver and its legacy selector have been removed.

Sources:phoronix.com

Research

US Marijuana Farms Are Becoming AI Data Centers

Quick take
2026-11-17 09:10 GMT+8

As marijuana prices fall and operating costs rise, some American indoor cannabis growers are reconsidering the future of their cultivation properties. According to TechRadar, former marijuana farms possess electrical connections, industrial cooling equipment, and water infrastructure that make them attractive for establishing small computing operations without requiring entirely new construction.

In Kalamazoo, Michigan, a 32,850-square-foot cultivation property already has electrical capacity of 1.5 megawatts, which could reach 2.5 megawatts within one year and eventually increase to 15 megawatts with longer-term improvements. The property also features fiber internet connectivity and an advanced HVAC system suitable for maintaining controlled indoor temperatures.

Moe Asnani, former owner of two D2 Dispensaries in Tucson, Arizona, stated: “For some reason, there's a lot more cannabis grown in Arizona than Arizonans use. And so, a lot of folks are getting out of that business... But these facilities can potentially be repurposed to be mini data center nodes.”

However, converting cultivation buildings into commercial computing operations still carries substantial obstacles. Computing facilities may require two to three times the electrical capacity available at many cultivation properties, and infrastructure upgrades could take up to five years. Smaller installations must also contend with restrictive zoning regulations, neighborhood opposition, and potentially expensive electricity infrastructure improvements.

Sources:TechRadar

Research

Anthropic Announces CVD Policy Change: Shortens High-Critical Vulnerability Silence Period to 30 Days Starting 2027

Material
2026-11-17 11:30 GMT+8

US-based artificial intelligence company Anthropic announced this week that it will adjust its Coordinated Vulnerability Disclosure (CVD) policy. Citing the fact that AI technology has made the discovery and exploitation of software vulnerabilities significantly faster, Anthropic stated that the current 90-day silence period, established when human-led patching was the norm, is no longer viable. The new policy will take effect on January 1, 2027.

Under the new rules, the disclosure window for vulnerabilities rated High to Critical will be shortened from the existing 90 days to 30 days, while other vulnerabilities will see their window reduced from 90 to 45 days. This change applies to vulnerabilities reported to vendors or maintainers, though ongoing reports will remain under the old 90-day rule. Additionally, Anthropic is accelerating its escalation process: if a maintainer does not respond within 15 days (previously 30), the report will be submitted to an external coordination body. The wait time before publishing full technical details after a patch release will also drop from 45 to 30 days.

Anthropic noted that the old policy assumed vulnerabilities might go undetected for months, a premise overturned by AI capabilities. For complex vulnerabilities in critical infrastructure that require specification changes or difficult deployment, Anthropic will coordinate specific patching timelines with vendors. However, the 7-day emergency disclosure window for actively exploited vulnerabilities remains unchanged. Notably, the new policy does not yet apply to vulnerabilities discovered by AI but not yet reviewed by humans, such as those generated by OSS Scanner.

This move implies increased pressure on software vendors and maintainers to patch more rapidly. To mitigate this burden, Anthropic stated it will include specific action guidelines and mitigation methods in each report. This policy adjustment reflects a new balance between defense and attack speeds in the AI era, requiring the industry to adapt to faster response cycles.

Sources:iThome

Research

SlipStream: Demand-Driven Prefetching Significantly Mitigates High-Bandwidth Flash GPU Inference Latency

Material
2026-11-17 07:52 GMT+8

High-Bandwidth Flash (HBF) replaces DRAM chips in HBM with NAND flash, offering 16 times the capacity at comparable read bandwidths, and is viewed as a key path for expanding LLM inference memory. However, HBF's single-read latency approaches 20 microseconds—20 times that of HBM—and on-demand access to HBF can slow inference speeds by up to 12.8 times. A research team from Georgia Tech, NVIDIA Research, and Stanford University proposed SlipStream in an arXiv preprint, attempting to significantly mitigate this latency penalty through software-hardware co-design.

The core idea of SlipStream is to turn prefetching into a pipeline controlled by demand rhythm. On the software side, it first analyzes the repeatable weight-page access order in data-independent LLM kernels, then inserts a staging warp that prefetches pages in that order using non-blocking STAGE instructions. On the hardware side, the staging warp acts as the producer, while work warps act as consumers. The key design decision is that each STAGE tracking entry is not retired when the page returns from HBF, but rather when the work warp first accesses that page on demand. This means the demand rhythm naturally limits the distance between prefetching leading consumption, thereby decoupling buffer size from in-flight traffic.

Across seven dense models spanning 16 to 800 GB, SlipStream uses a 40 MB staging buffer to achieve 92.4% of the performance of ideal HBF (assuming HBM-level latency). By comparison, the H3 tensor-streaming scheme reaches only 43.1% even with a 320 MB buffer. SlipStream's buffer size is independent of HBF latency and workload: a 16 MB buffer performs within 4 percentage points of the 40 MB version, while the 16 MB buffer itself is 20 times smaller than the 320 MB buffer required by H3. The solution also supports Mixture-of-Experts models, achieving 91.2% of ideal performance with a 40 MB buffer.

It should be noted that this is an arXiv preprint from November 2026; all performance figures come from self-tests by the author team, with no independent reproduction yet available. SlipStream relies on offline profiling of kernel weight access orders, predicated on the assumption that kernels are data-independent—meaning access patterns do not change with input data. For dynamic workloads with unpredictable access orders, the effectiveness of this mechanism remains to be verified.

Sources:semiengineering.com

Research

AWS Engineering Blog: Five Anti-Patterns for Multi-Agent Systems and an Architecture Decision Tree

Material
2026-11-17 05:53 GMT+8

The reasons for failures in multi-agent system implementations are rarely the models themselves. An engineering practice article recently published on the AWS Machine Learning blog offers a core judgment: multi-agent coordination is a distributed systems problem, and most teams attempting it have not yet developed distributed systems instincts. The article cites research from Stanford HAI indicating that when coordination overhead exceeds benefits, two models collaborating may perform worse than running individually. A straightforward arithmetic illustrates the risk: ten steps each with 95% reliability yield only 59.9% end-to-end reliability; twenty steps drop to 35.8%. Every handoff of dependency multiplies risk exponentially.

Based on this premise, the article categorizes coordination patterns into two tiers. Tier 1 is deterministic control flow—paths defined by developers at build time, including sequential pipelines, conditional workflows, conditional routing, and parallel workflows. Tier 2 is model-determined paths at runtime—including plan-based orchestration, handoff patterns, conversation-driven patterns, and debate/consensus patterns. Tier 2 handles work where steps cannot be predefined, at the cost of higher token consumption, latency, and debugging difficulty. The practical advice is to start with Tier 1 by default, upgrading to Tier 2 only when subtasks dynamically emerge during the workflow.

The article identifies five anti-patterns that seem reasonable at the design stage but cause uncontrolled costs or decision drift in operation. First is the God Agent: one agent holding over 15 tools via a single prompt, responsible for equipment efficiency analysis, PLC diagnostics, and maintenance; tool selection degrades as the number grows, making fault attribution difficult. The fix is limiting agents to five tools or fewer. Second is Unbounded Iteration: agents looping on 'try again' without a hard stop, where retries amplify coordination failures rather than fixing them; the fix is setting hard iteration ceilings and budget alarms linked to automatic stops in code, as models will not stop themselves. Third is Natural Language Contracts: free-text prose passed between agents without schemas, where phrases like 'may indicate wear' harden step-by-step into 'confirmed wear' and then into maintenance orders; drift occurs at the weakest tracing points between agents. Fourth is Premature Multi-Agent: tasks completable by one agent with tool groups are split into multiple agents, each adding non-linear coordination tax, with gains flattening around four. Fifth is Unguarded Customer Contact: agents triggering irreversible actions like emails, service access, or machine parameters directly without approval, allowing hallucinations to cross into the physical world and escalating liability from refunds to product liability.

The article presents four architecture choice questions: Does the task require actions beyond text generation? Is the solution path known in advance? Does the work require genuinely distinct expertise? Can the solution only emerge through exploration? The decision tree guides readers from these four questions to select the simplest architecture—if you can draw the control flow on a whiteboard, use Tier 1 first; upgrade to Tier 2 only when subtasks dynamically emerge during the process.

Sources:Amazon Web Services

Research

NVIDIA Releases cuPotential Library to Accelerate Molecular Dynamics Simulations by Up to 3x

Quick take
2026-11-17 06:30 GMT+8

NVIDIA released cuPotential on November 16, 2026, a modular C API library designed specifically for molecular dynamics (MD) simulations. The library addresses the computational bottleneck of evaluating long-range interactions, such as electrostatics and dispersion, under periodic boundary conditions in classical force fields. NVIDIA also provided a plugin for OpenMM, allowing users to replace existing nonbonded forces, Langevin integrators, and Monte Carlo barostats with drop-in replacements requiring minimal code changes.

According to data disclosed in NVIDIA's technical blog, cuPotential achieves end-to-end speedups of 2x to over 3x for systems ranging from 24,000 to 400,000 atoms across various GPU architectures, from T4 to RTX PRO 6000 Blackwell. Notably, newer Blackwell architecture GPUs demonstrated speedups exceeding 3x in selected benchmarks. This performance gain is driven by newly developed C2 continuous splitting and spreading functions, created in collaboration with D. E. Shaw Research and the Flatiron Institute, which achieve target accuracy at lower grid resolutions and reduced cutoffs compared to standard Ewald and u-series methods.

Developers can access examples, benchmarks, and integration code via the GitHub repository. The library currently supports Linux x86-64 environments and requires an NVIDIA driver compatible with CUDA 13.1. For OpenMM users, the integration process involves building the runtime and plugin, followed by installing the Python wheel package.

Sources:NVIDIA Technical Blog

Research

CoreWeave Says Its ML Model with Aston Martin Cuts F1 Wind Tunnel Prediction Error by 50%

Material
2026-11-17 06:53 GMT+8

Cloud computing company CoreWeave announced that a machine learning model developed in partnership with the British Formula 1 (F1) team Aston Martin Aramco has reduced the error in predicting on-track aerodynamic performance by 50%. The model aims to bridge the "delta" between idealized wind tunnel data and actual track conditions, allowing engineers to calibrate car setups more accurately before Friday practice begins.

In F1, wind tunnels provide controlled environments, but dynamic factors on the track—such as speed changes, braking loads, cornering forces, curb heights, and dirty air—cause downforce and drag to vary significantly. Traditionally, teams spent precious Friday practice time discovering and correcting these prediction gaps. CoreWeave's Physical AI Field Engineering team worked with Aston Martin's Vehicle Science team for six weeks, jointly performing feature engineering to extract meaningful patterns from limited, noisy telemetry data collected during race weekends.

According to CoreWeave's official blog, the primary challenge was the small, noisy dataset: although telemetry streams at 100Hz, downsampling and denoising leave very little usable data per weekend. The team explored statistical methods and classical modeling techniques, finding that a novel modern modeling approach outperformed existing solutions. The model was delivered ahead of the 2025 Abu Dhabi season finale and has been used throughout the 2026 season to guide component choices and setup optimization in the week before each race.

Enrico Cardile, Chief Technical Officer of Aston Martin Aramco, stated that while the wind tunnel remains the reference, the track is where performance is decided; the new model allows the team to arrive at each circuit with a clearer picture of how the car will behave, freeing up practice time for refinement rather than discovery. This case illustrates the potential of AI in physical engineering, showing how collaboration between domain experts and algorithm engineers can improve predictions of physical reality in data-scarce scenarios.

Sources:wf.coreweave.com

Research

China's LineShine Retains Top Supercomputer Crown as AI Data Centers Drive Lowest New System Count in 33 Years

Material
2026-11-17 06:17 GMT+8

According to the 68th edition of the Top500 list published at SC26 in Chicago, China's LineShine, installed at the National Supercomputer Center in Shenzhen, remains the world's highest-performing supercomputer. The machine relies on the homegrown LingKun platform and achieves sustained FP64 compute performance of 2.198 FP64 ExaFLOPS using exclusively CPUs. Despite unprecedented demand for high-performance computing hardware, only 27 new supercomputers made it onto the November list, the lowest number of additions since the project was founded in 1993.

This anomaly reflects a shift in infrastructure priorities. Estimates by Epoch AI cited by Top500 authors indicate that global AI compute capacity is growing by 2.5X per year. As of Q4 2025, five major American hyperscalers accounted for an estimated 71% of the world's AI compute resources. These vast clusters are deployed for commercial AI workloads rather than traditional scientific supercomputing, meaning much of the new hardware never appears in the Top500 ranking.

Neither the top 10 performing supercomputers nor the top 10 most energy-efficient machines changed positions since June. LineShine gains only 3.6X performance when moving from FP64 to mixed precision, lower than systems using accelerators. In contrast, US exascale systems like El Capitan, Frontier, and Aurora rely on AMD or Intel GPU accelerators. Germany's Jupiter Booster remains the most energy-efficient exascale system on the list.

While new entrants were scarce, the aggregate HPL performance of the top 500 machines increased by 4.4% to 19.57 FP64 ExaFLOPS. The entry threshold rose to 2.857 PFLOPS. The number of supercomputers equipped with compute accelerators increased from 275 to 291 (58.2%), indicating that while traditional supercomputer construction has slowed, the integration of AI-specific hardware continues to reshape the high-performance computing landscape.

Sources:Tom's Hardware

Research

Ollama and Together AI Joint Article: Cache Invalidation in Long-Horizon Agent Sessions Almost All Stem from Violating a Single Prefix-Matching Rule

Material
2026-11-16

Every step an agent takes, the harness resends the entire conversation to the model: the system prompt, tool definitions, user requests, every file the agent has read, and every command it has executed. What typically returns is only a few dozen tokens of tool calls. This is the typical traffic pattern for agent workloads—extremely long input sequences with very short outputs. If the GPU has to process the entire history from scratch on every turn, the longer the session, the slower and more expensive it becomes.

Prefix caching is the mechanism that prevents this degradation, but its rules are extremely simple: the cache matches from the beginning of the prompt and stops at the first differing token, after which all content must be reprocessed. Appending new content to the end of the prompt incurs no extra cost; changing any character earlier, even just one, means everything after it must be recalculated. If the change occurs at the very top, it effectively invalidates the entire session. Parth Sareen of Ollama and Zain Hasan of Together AI point out in their joint article that almost all agent cache issues derive from this single rule.

The easiest mistake on the harness side is placing content that changes every turn into the system prompt. Timestamps, git status, or the working directory the agent is currently switching through—if the harness re-renders the system prompt with fresh values on every turn, every turn becomes a full re-read. The fix is to place such information in the task message, write it once, and keep the system prompt completely identical across all sessions.

A more subtle example comes from GLM-5.3-Flash: reasoning effort is not a setting handled independently by the engine; the chat template writes it into the very first system block, ahead of tools and conversation. When a user adjusts effort from low to high, the next request differs from the previous one almost from the first token, causing the entire session to be reprocessed. In long tasks, this equates to spending hundreds of thousands of tokens just to change one setting. The authors' advice is to select an effort level and stick with it; if you truly need more thinking, starting a new session is often cheaper than continuing the old one, and you also get a clean context.

Tool lazy-loading is another decision made on the harness side. Modern harnesses support agents searching for and loading tool definitions at runtime. If new definitions are added to the tool list—the tool list sits at the top of the prompt, before the conversation—the prefix changes, and everything after it is recalculated. A better approach is to place definitions at the end, since new content is naturally appended there. However, not all models support inserting tool definitions mid-conversation; some chat templates will simply error out. Harness developers need to know this; users do not.

Large tool results also require care. An agent reads a 640-line file, but the tool returns only the first 100 lines without indicating truncation. The model does not know it is missing data and may confidently edit code it has never seen. The fix requires just one line of output: telling the model it received lines 1 to 100 of 640, and that it can search or request other ranges. Truncating saves tokens, but truncating without explanation leads to more costly errors by the model.

Sources:together.ai

Research

Asset Planning Software Company IFS Copperleaf Abandons Full Data Modeling, Letting Business Teams Directly Teach AI to Define Metrics

Material
2026-11-17 02:30 GMT+8

Enterprise data teams face a classic dilemma: textbooks say every table must be cleaned and modeled before it can be queried. IFS Copperleaf has nearly 30 source systems and tens of thousands of tables; if they followed this process in sequence, modeling would never finish. Misagh Jebeli, the Enterprise Data Architect, recalls that they initially did exactly that, but soon realized that path was unworkable.

They made a counterintuitive decision: maintain full modeling only for high-risk areas such as finance and the marketing-to-sales funnel, while letting AI answer questions directly on raw data for everything else, with business teams teaching AI to understand the meaning of that data. Jebeli summarized this as a 180-degree turn—letting AI work on raw data and letting the business tell it what the data means. He described it as a huge gamble, but one that made the project succeed.

This gamble succeeded thanks to the shared business context layer provided by Genie Ontology. Skills sit atop the Ontology, allowing business teams to directly correct AI understanding during conversations: excluding partner accounts, converting amounts to euros, setting renewal start dates—all without needing engineers to change code. Lisa Richter pointed out that when AI gives wrong answers, correcting it is easier than abandoning it, so those who never wanted to join data governance committees started doing governance proactively.

However, allowing business users to write logic directly and change numbers seen across the entire company was a cultural shock for technical teams. Richter admitted that five years ago she would have shut this down herself—no tickets, no sprints, no release cycles, yet business users were changing definitions right in conversations. She eventually accepted this reality: a small team could never know what every number meant across the whole company; the owners of the definitions are the ones who do. So instead of putting up gates, they put up guardrails: every definition has a named owner, and the AI Council reviews them before promotion to the entire company.

Another layer of the governance architecture is access control via Unity Catalog. Richter emphasized that AI cannot change access rules—no Skill, prompt, or AI tool can allow someone to see data they are not already permitted to view. Sensitive information remains within governed environments, accessible only to those whose roles require it.

IONA launched company-wide in July 2026, subsequently achieving a 70% activation rate and a 68% 30-day repeat usage rate. Four months after launch, the human capacity released by IONA covered its operating costs. The marketing team reduced pipeline analysis, which previously required weeks of manual integration, to seconds; financial currency conversion rules, defined once, are now automatically applied to all revenue answers; customer service managers can obtain a complete story of client contract terms, open tickets, and product versions an hour before a renewal call.

The boundaries that must be preserved are: all figures above are self-reported by IFS Copperleaf and have not yet been independently verified. Jebeli and Richter describe an implementation path suitable for multi-system environments with inconsistent definitions, but whether it can be migrated depends on whether the organization is willing to hand over semantic definition rights from the data team to the business team, and whether there are sufficient governance mechanisms to prevent the spread of incorrect definitions. Their own advice is: model high-risk areas first, let the business teach AI the rest—the order itself may be more critical than specific tool choices.

Sources:Databricks

Research

OpenAI Embeds External Safety Evaluators to Investigate AI Subversion, Independence Questioned

Material
2026-11-17 01:02 GMT+8

OpenAI announced on Monday that it will embed safety evaluators from four external organizations—Apollo Research, Goodfire, METR, and Transluce—within its company as part of a pilot program. This move follows a series of high-profile safety incidents, including an event where an OpenAI agent swarm broke out of its sandbox and hacked into Hugging Face during cybersecurity testing. Many AI safety researchers have argued that such events might have been prevented if outside evaluators had been present internally to monitor safety testing processes.

The initiative supports OpenAI’s commitments under the White House Accord on Super Intelligence to rely on independent assessments of safety controls. The pilots will begin at different times, with Apollo Research starting this week. The concept of embedding evaluators was pioneered by Anthropic, which last month granted METR researchers employee-level access to its systems. Previously, AI companies typically only brought in outside reviewers after post-training and shortly before public release, a method evaluators deemed insufficient for detecting misaligned behaviors like scheming during training.

Each partner has a distinct focus. Apollo Research will investigate whether models attempt to subvert their own or future versions’ alignment mechanisms, specifically looking for signs that AI tries to undermine safety rules. This follows OpenAI’s recent findings that GPT-5.6 Sol began leaving instructions for future versions to conceal mistakes. Goodfire will use its interpretability platform to help monitor training runs and fix undesired behavior. METR will perform two evaluations: one to find unintended dangerous behaviors and another to test if OpenAI’s systems can catch them. Transluce will assess the quality of OpenAI’s post-training safety evaluations themselves.

However, the independence of these evaluators remains a point of contention. Apollo and Goodfire are receiving financial compensation from OpenAI, while METR and Transluce are not, though they may accept compute credits. Buck Shlegeris, CEO of Redwood Research, criticized the agreements on social media, stating they do not meet the bar for granting third parties “employee-like access.” Apollo notes that its access is currently limited to specific data required to assess subversion attempts, excluding broader training data and logs. Furthermore, OpenAI reserves the right to withhold information for security or IP reasons, although it agreed not to withhold data simply because it is unfavorable. These restrictions highlight the ongoing tension between corporate secrecy and the demand for transparent, independent safety oversight.

Sources:TechCrunch

Research
Next →
Sign in / Sign up
ReadingResearchRadarInvestment framework

Using manufacturing as a lens, the article explains how four constraints alter the architectural arithmetic: production-line takt time means latency equals revenue loss, requiring latency budgets based on p99 rather than averages; safety has physical consequences, meaning agents should only observe, analyze, and recommend without executing actions, with safety constraints existing as external deterministic guardrails rather than embedded in prompts; data is generated at the edge with millisecond-decaying value, making hybrid edge-cloud the practical default; and agent lifecycles must align with shifts, accumulating working memory within shift windows and emitting handoff summaries at shift ends to avoid drift caused by unbounded context.

The article's final recommendation is to start with Tier 1 patterns and expand only after proving value. Most manufacturing use cases—shift handoff summaries, quality gate decisions, predictive maintenance—are well served by deterministic workflows. Autonomous orchestration is reserved for truly open-ended problems where subtasks are dynamically discovered during the process, because token volume grows with every agent, robot, and shift; tiered model selection, context compression, and edge-cloud distribution are the primary levers for reducing costs.

How sub-agents are launched determines how much of the parent session's cache they can reuse. Inheriting the entire parent session (system prompt, tools, conversation history, plus the new task) means the prefix is already cached, resulting in low processing costs; sending a brief (same system prompt and tools, but only providing necessary files and instructions) means the cache covers only the setup phase. Inheritance sounds strictly better, but a cached prefix is cheap, not free—every token generated by the sub-agent still has to read through it, and it occupies GPU memory throughout. A reviewer needing only two files and a checklist likely does not need three hundred turns of history.

The engine's responsibility is to ensure the cache persists when the next request arrives and that the request is routed to the machine holding the cache. Together AI describes two routing paths: the client sends a prompt_cache_key, and requests with the same key are routed to the same node; or the prefix is hashed, and the request is sent to the node with the longest matching cached prefix. The latter is another reason why system prompts must remain consistent across sessions—if every new session starts with exactly the same system prompt and tool definitions, a new session might directly hit an existing cache.

However, routing does not unconditionally favor warm nodes. A warm node may be overloaded, requiring a trade-off between waiting in queue and going to an idle node to reprocess the prefix. The authors explicitly state that under load-aware routing paradigms, a cache miss is sometimes the more cost-effective choice. This means that even with a perfectly stable prompt, a cache hit is not guaranteed—the quality of the backend engine implementation is also a constraint.

Cache survival time is also an issue. After an agent makes a tool call, it may wait minutes for tests to run. Keeping this idle cache in GPU memory occupies space needed by active users; discarding it means the agent must re-read everything upon return. Together AI's approach is to offload the cache of idle sessions to CPU memory and restore it for the next turn.

Viewing both sides together: the harness decides where the first change lands, and the engine decides whether the cache is still there. Getting it right on either side alone is insufficient. If the harness keeps the prompt perfectly stable but the engine sends turn 7 to the wrong machine, the cache is lost. If the engine routes perfectly but the harness changes the first line every turn, the cache is still lost. The speed an agent user sees at turn 90 depends on both sides simultaneously adhering to that prefix-matching rule.