Salesforce Cross-Region Inference Routing Test: Latency Spikes Do Not Trigger Failover; Only Errors Shift Traffic
The Agentforce team at Salesforce deploys models across more than 30 regions, previously relying on a self-built gateway for cross-region disaster recovery. After integrating the Amazon SageMaker AI cross-region inference router, they observed counterintuitive behavior: when local endpoints became saturated and p50 latency spiked from 0.2 seconds to 58 seconds, the router kept traffic in place; traffic only shifted to remote regions after requests began returning errors, with the transition occurring gradually over approximately three minutes. The article also explains the architectural separation where data residency checks remain at the application layer while capacity routing is handled by the platform layer.
The Agentforce Foundation team at Salesforce provides model inference services for products such as Agentforce and the Trust Layer across more than 30 global regions, with each region's endpoint independently handling local traffic. The challenge lies in the fact that peak usage windows for different customers often overlap; provisioning capacity based on overlapping peaks is prohibitively expensive, while scaling out region by region leads to fragmentation—some regions have idle instances while others are fully utilized. Before integrating the SageMaker AI cross-region inference router, Salesforce relied on its self-built Internal Gateway to manually shift traffic that local endpoints could not handle to other regions, supported by basic health checks, retries, and region-switching logic. This approach had two gaps: the gateway lacked capacity signals from either local or remote endpoints, making it unable to determine when to switch traffic or whether the destination had available capacity; additionally, cross-region traffic had to exit the VPC through NAT gateways and internet gateways, traversing the public network.
The router addresses the first gap by enabling the routing layer to perceive the health status and remaining capacity of each endpoint directly, eliminating the need for callers to guess when to switch. During integration, Salesforce simultaneously implemented a critical architectural separation: data residency checks remained at the gateway, while capacity routing was delegated to the platform. Requests tagged with residency markers are pinned to a routing target that lists only local endpoints as destinations, ensuring they never cross region boundaries; all other requests go through the router, with request bodies, content types, and accept types remaining identical to direct InvokeEndpoint calls. To address the second gap, Salesforce used VPC endpoints (AWS PrivateLink) to keep the call from the gateway to the router within the private network, bypassing NAT gateways and the public internet. The router itself is an account-level regional resource; its routing targets list endpoints capable of serving a specific logical model using ARNs, which can point to another region or even another AWS account; endpoint owners authorize specific routers via resource policies and can revoke access at any time.