Last reviewed: August 4, 2026

Direct answer

Treat data residency as a routing precondition, not a preference that can be relaxed during an outage. Classify each request before it reaches an LLM provider, translate that classification into allowed processing locations, storage and retention rules, and tool restrictions, then remove every ineligible route from consideration. Only after that filter should the router compare health, capacity, latency, or cost.

If the primary route fails and another healthy route satisfies the same contract, fail over to it. If no eligible route remains, fail closed or enter a preapproved local degradation mode. Do not silently widen an EU zone to a global deployment, replace a single-region route with a geographic route, or assume that data stored in one place was also processed there.

This ordering is the central rule for LLM API failover data residency:

request policy -> eligible routes -> healthy routes -> selected route

Availability logic may narrow an eligible set. It must never expand that set.

Who this is for

This design is for platform engineers, SREs, security engineers, privacy owners, and AI infrastructure teams that route prompts across providers, accounts, deployments, or regions. It matters most when different tenants or workloads have different boundaries, such as one workload restricted to a single region, another allowed anywhere inside a named geography, and a third with no geographic restriction.

It also applies when an application can invoke retrieval systems or external tools. The LLM endpoint may be eligible while a connected tool is not, so the route contract has to cover the complete request path rather than the model call alone.

Key takeaways

  • Attach a residency policy to the user action before provider selection. A tenant default can be useful, but the effective policy should travel with the request and be immutable for that action.
  • Model provider routes as verified capabilities. A provider name or model alias is not enough; deployment type, configured region, possible processing scope, storage behavior, retention profile, and external egress all matter.
  • Filter for compliance before checking health or price. A fast global endpoint is not a fallback for a region-bound request unless the policy explicitly permits global processing.
  • Make an empty eligible set an expected operational state. Return a stable error or use a specifically approved degradation path instead of improvising during an incident.
  • Log the decision and its policy evidence without logging prompts, responses, personal data, request headers, or authentication material.
  • Revalidate the route catalog when models, deployment types, regions, retention controls, or connected tools change.

The policy should also stay bound to the original action across retries and asynchronous work. The approach in binding fallback decisions to the user action prevents a retry worker from reconstructing a weaker policy from incomplete queue metadata.

Sources checked

The OpenAI data controls documentation separates API data into categories such as abuse-monitoring logs and application state. It says abuse-monitoring logs may contain prompts and responses, are retained for up to 30 days by default, and that eligible customers need approval for alternative controls. That is a useful reminder that an inference-location rule is not a complete data contract: storage and retention must be checked independently.

The Microsoft Foundry deployment-type documentation makes processing scope an explicit property of the deployment type. Its current description says global types may process inference data in any Azure region, Data Zone types process within the specified US, EU, or APAC zone, and standard or regional types process in the deployment region. It also warns that not every model supports every deployment type. Therefore, a route catalog must identify a concrete deployment and model combination rather than infer residency from the provider account’s home region.

The Amazon Bedrock cross-Region inference guide distinguishes geographic profiles, which route within a named geography, from global profiles, which can route worldwide across supported commercial Regions. It recommends geographic profiles for data-residency requirements. The guide also identifies additionalEventData.inferenceRegion in CloudTrail as evidence of where a cross-Region request was processed. That field is useful for verification, but it should confirm an already eligible choice rather than serve as permission after a request has been sent.

Together, these sources show why a generic label such as multi-region is too vague. Provider mechanisms draw different boundaries and expose different evidence. Your internal policy must normalize those differences without pretending they are identical.

Contract details to verify

Build a versioned eligibility catalog for every callable route. At minimum, verify:

  • the provider, account or project, concrete deployment, model, and model version;
  • the configured source region and every location in which inference may occur;
  • whether the processing boundary is a single region, named zone or geography, or global;
  • where application state and operational logs may be stored, and the applicable retention control;
  • whether batch, streaming, caching, retrieval, and tool calls follow the same boundary;
  • whether connected tools or data stores introduce another processing location;
  • the source used to verify each assertion, the review date, and the catalog version that approved it.

Do not reduce this to one region column. Keep the requested boundary and the route’s effective processing scope separate. A route configured in an EU region can still be ineligible if its deployment type permits global processing. Conversely, a geographic profile can be valid for an EU-wide policy but not for a contract requiring one particular region.

Use an explicit operator workflow for both outcomes.

Happy path

  1. A request arrives with the immutable policy class eu-zone and tool use disabled.
  2. The router loads the active policy and route-catalog versions, then filters out global, single-region-outside-EU, stale, and tool-dependent routes.
  3. The primary eligible route is unhealthy. A second route whose documented processing scope is the EU remains healthy and has capacity.
  4. The router selects that route, records the reason, and sends the request once within the action’s retry budget.
  5. The response is accepted only if the route identity still matches the evaluated catalog entry. Available processing-region evidence is recorded and checked for an unexpected value.
  6. Operators monitor both service health and residency-policy denials while the fallback is active. Return to the primary follows a controlled recovery decision, not an automatic widening of traffic.

Error path

  1. A single-region request arrives while its primary is unavailable.
  2. The only healthy alternatives are global, are outside the allowed region, or have stale eligibility evidence. Filtering produces zero routes.
  3. The router does not send the prompt. It returns a stable residency_route_unavailable result, or queues the work only if that queue and its retention period are approved for this policy.
  4. The operator confirms the request policy, catalog version, rejection reasons, and primary health. The operator does not manually enable a global route to clear the queue.
  5. Recovery requires restoring an approved route, verifying a new route through the normal change process, or invoking a documented local degradation mode. Affected users receive a service-availability message that does not expose internal topology.

Use structured, sanitized logs so this workflow can be reconstructed:

{
  "event": "llm_route_decision",
  "request_ref": "req-42",
  "policy_class": "eu-zone",
  "policy_version": "v4",
  "catalog_version": "v7",
  "candidate_count": 3,
  "eligible_route_refs": ["route-eu-a", "route-eu-b"],
  "selected_route_ref": "route-eu-b",
  "selection_reason": "primary_unhealthy",
  "deployment_scope": "eu-zone",
  "configured_region": "eu",
  "observed_inference_region": "eu-west",
  "retention_profile": "profile-a",
  "external_tool_egress": false,
  "decision": "routed",
  "status_code": 200
}

For a denial, set the selected route to null, record categorical rejection counts such as scope_mismatch or evidence_stale, and record the stable application error. Do not put raw prompts, outputs, document contents, user identifiers, headers, cookies, or provider error bodies in this event. Sanitize location and route fields against controlled values so untrusted text cannot enter logs.

The same record should feed incident evidence. The guidance on fallback decision logs provides a broader review pattern for explaining why a route was selected.

Failure modes

A global route hides inside the fallback list. A simple ordered list advances from a regional primary to a global secondary after a timeout. Prevent this by generating the list from the request policy and route catalog, not by maintaining one universal provider order.

Storage geography is mistaken for inference geography. A control-plane resource or stored data may remain in one geography while inference uses a broader processing scope. Represent and test those properties separately.

The catalog drifts from the deployed system. A model upgrade, deployment-type change, renamed alias, or newly connected tool can invalidate earlier evidence. Reject unknown combinations by default and make configuration deployment contingent on a successful catalog check.

A retry or hedge escapes the original policy. A worker may rebuild context from tenant defaults, or two concurrent attempts may use different route sets. Persist the effective policy and catalog version with the action, cap attempts, and cancel losing work. Review attempt limits per user action before combining retries with failover.

Telemetry is treated as authorization. Discovering an unexpected inference region after completion is evidence of a control failure, not a substitute for pre-routing enforcement. Quarantine the route, stop further eligible traffic to it, preserve sanitized evidence, and assess affected requests.

The no-route error triggers another generic fallback layer. An application proxy may interpret every upstream error as retryable and send the same content elsewhere. Give policy denials a distinct, non-generic error code and ensure every layer recognizes it as a prohibition on geographic expansion.

Logs become a second data path. Prompt excerpts and raw provider errors can copy restricted content into an observability system with different storage rules. Use references and controlled categories, and apply the same residency and retention review to the logging pipeline.

FAQ

Is keeping data at rest in a region enough?

No. Processing location, application state, operational logs, retention, and external tool egress are separate contract dimensions. Verify each one for the exact route and features in use.

Can a global deployment be an emergency backup?

Only for requests whose policy explicitly allows its full processing scope. An incident does not create consent to widen a boundary. Restricted requests should receive the approved degradation or no-route outcome.

Can redaction make an otherwise ineligible route safe?

Only when an approved policy defines the transformation, tests show it removes all restricted data needed by that policy, and the reduced request still serves the user’s intent. Redaction should not be an improvised incident bypass.

What if the provider does not report the processing region per request?

Use documented deployment guarantees, configuration evidence, and provider-supported audit records to establish eligibility before routing. Record the absence of request-level evidence as an evidence limitation; do not invent precision or infer location from latency.

How should operators test the fail-closed path?

In staging, mark every route for a restrictive policy unhealthy or ineligible. Confirm that no provider receives the request, the stable policy-denial result reaches the application, queue behavior matches retention rules, and the sanitized decision event contains enough information to explain the outcome.

Reader next step

Create a route-eligibility table for one production workload today. Give each concrete deployment separate processing-scope, storage, retention, tool-egress, evidence-source, and last-verified fields. Then run two tests: disable the primary while a compliant secondary is available, and disable every compliant route while a healthy global route remains. The first request should move; the second must not.

Turn the results into a runbook entry with an owner and revalidation trigger. That gives on-call engineers a reliable decision boundary before the next provider incident makes a global endpoint look tempting.