ReadingResearchRadarInvestment framework
Sign in / Sign up
← Back to reading
NewsMaterial4 min readDisconfirmAI Editorial

Salesforce Cross-Region Inference Routing Test: Latency Spikes Do Not Trigger Failover; Only Errors Shift Traffic

The Agentforce team at Salesforce deploys models across more than 30 regions, previously relying on a self-built gateway for cross-region disaster recovery. After integrating the Amazon SageMaker AI cross-region inference router, they observed counterintuitive behavior: when local endpoints became saturated and p50 latency spiked from 0.2 seconds to 58 seconds, the router kept traffic in place; traffic only shifted to remote regions after requests began returning errors, with the transition occurring gradually over approximately three minutes. The article also explains the architectural separation where data residency checks remain at the application layer while capacity routing is handled by the platform layer.

2026-11-19 04:13 GMT+8

The Agentforce Foundation team at Salesforce provides model inference services for products such as Agentforce and the Trust Layer across more than 30 global regions, with each region's endpoint independently handling local traffic. The challenge lies in the fact that peak usage windows for different customers often overlap; provisioning capacity based on overlapping peaks is prohibitively expensive, while scaling out region by region leads to fragmentation—some regions have idle instances while others are fully utilized. Before integrating the SageMaker AI cross-region inference router, Salesforce relied on its self-built Internal Gateway to manually shift traffic that local endpoints could not handle to other regions, supported by basic health checks, retries, and region-switching logic. This approach had two gaps: the gateway lacked capacity signals from either local or remote endpoints, making it unable to determine when to switch traffic or whether the destination had available capacity; additionally, cross-region traffic had to exit the VPC through NAT gateways and internet gateways, traversing the public network.

The router addresses the first gap by enabling the routing layer to perceive the health status and remaining capacity of each endpoint directly, eliminating the need for callers to guess when to switch. During integration, Salesforce simultaneously implemented a critical architectural separation: data residency checks remained at the gateway, while capacity routing was delegated to the platform. Requests tagged with residency markers are pinned to a routing target that lists only local endpoints as destinations, ensuring they never cross region boundaries; all other requests go through the router, with request bodies, content types, and accept types remaining identical to direct InvokeEndpoint calls. To address the second gap, Salesforce used VPC endpoints (AWS PrivateLink) to keep the call from the gateway to the router within the private network, bypassing NAT gateways and the public internet. The router itself is an account-level regional resource; its routing targets list endpoints capable of serving a specific logical model using ARNs, which can point to another region or even another AWS account; endpoint owners authorize specific routers via resource policies and can revoke access at any time.

Sign in / Sign up
ReadingResearchRadarInvestment framework

What truly caught the Salesforce team's attention during testing was the router's counterintuitive health assessment logic. The router evaluates each destination based on recent request outcomes: platform faults, 5xx errors returned by the model container, throttling, and timeouts reduce a destination's health score. However, model latency alone does not trigger failover—a slow response often reflects long prompts or outputs and does not necessarily indicate an unhealthy endpoint. This design choice produced specific consequences during Salesforce's saturation tests: they saturated the local endpoint in us-west-2, causing p50 latency to rise from 0.2 seconds to between 45 and 58 seconds for about five minutes, yet traffic remained completely stationary. It was only when requests began exceeding the model container's 60-second handler timeout and returning errors that the router started shifting traffic; the first batch of requests reached us-east-2 about two minutes after the initial errors, and within another minute, approximately 78% of probe traffic had switched to the remote region. The remaining 20% to 25% stayed local to allow the router to detect recovery.

Once traffic shifted, p50 latency returned to 0.2 seconds, with the cross-region path adding less than 100 milliseconds. After the local endpoint recovered, traffic slowly flowed back over approximately ten minutes, slower than the outbound shift to prevent oscillation. Throughout this process, no errors were recorded by probes. Salesforce treats the InvokedRoutedResourceArn and RoutingReason fields carried in every response as first-class telemetry, logging them on the same structured line as request IDs and trace IDs, and emitting them as APM tags. Because the ARN inherently contains region and account information, a single log line reveals whether a request executed locally or crossed a boundary, along with the reason: local-preferred, failover, capacity-available, or only-healthy.

This pattern has a clear prerequisite: the router does not create capacity, replicate model artifacts, or add instances. It utilizes already deployed capacity, with each endpoint's own autoscaling responsible for adding instances as traffic grows. Salesforce lists several items pending verification, including holding the local endpoint at saturation for longer periods, measuring cross-region latency with millisecond precision, and evaluating cascading behaviors at the cross-region traffic cap. These incomplete tests imply that the currently observed switching time windows and return rhythms may apply only to short-duration saturation scenarios.

For readers seeking transferable insights: if you are designing disaster recovery solutions for multi-region AI inference, using latency thresholds as a switching signal will prematurely trigger unnecessary cross-region traffic shifts, whereas relying solely on error signals will force local users to endure high latency during saturation windows that have not yet timed out. The trade-off depends on whether your SLOs are bound to response time or success rates. Salesforce's tests demonstrate that under a 60-second handler timeout constraint, a saturated endpoint can operate at 45 to 58 seconds p50 for five minutes without the router deeming it unhealthy—during this period, local user experience severely degrades, yet there is no change in platform-level traffic distribution.

Sources

  • aws.amazon.comAmazon Web Services↗
Continue researching →