Keep TLS Trust Intact During LLM API Failover

Last reviewed: 2026-10-16

Direct answer

Keep TLS trust intact during LLM API failover by treating the fallback connection as a new security boundary. Use a named hostname, a verified certificate chain, and hostname validation for every route. Select the trust store deliberately, preserve the logical server name when you connect through a pinned address or proxy, and fail closed when the alternate route cannot meet the same verification contract. A route switch may change the socket, region, or client pool; it must not turn certificate errors into permission to set verify=False, disable rejectUnauthorized, or accept an unreviewed CA.

A warm connection, a successful TCP handshake, or a DNS answer is transport evidence. None proves that the peer is the intended API service. curl’s SSL certificate documentation explains that a CA store is used to verify the server certificate and that curl also checks whether the certificate was created for the server name in the URL. HTTPX’s SSL guide makes the same boundary visible: HTTPS verification is on by default, while verify=False allows insecure requests. Treat those defaults as part of the route contract, rather than incidental library behavior.

The fallback contract should name the endpoint hostname, port, proxy path, trust-store source, minimum verification mode, server-name or SNI value, client-certificate requirement, and the error that stops promotion. For certificate or trust-store rotation, require a full, non-resumed TLS handshake against the candidate chain. A new socket alone can resume an earlier TLS session and retain its earlier authentication result. Keep socket reuse and session reuse as separate evidence fields.

Who this is for

This guide is for SREs, platform engineers, and SDK owners who operate LLM gateways with a primary route and one or more fallback routes. It applies to HTTP clients, sidecars, service meshes, and language runtimes that may each own part of DNS, TCP, TLS, proxying, or connection pooling. It is especially useful during a regional cutover, certificate rotation, egress migration, or emergency provider change.

The focus is the client-side verification contract. It does not claim that any provider supports a particular certificate chain, private endpoint, mTLS policy, or failover hostname. Verify those details for your account and deployment, then use the same evidence fields on every route.

Key takeaways

• Preserve the trust contract on every fallback connection; a new socket can still resume a previously authenticated TLS session.
• Keep hostname validation and the TLS server name aligned with the intended endpoint; an IP address used for dialing is not the identity to verify.
• Pin a deliberate CA source per runtime and record its version or deployment revision.
• Keep client-certificate material separate from the CA bundle; mTLS authenticates the client and does not replace server verification.
• Never turn off verification as a failover workaround. Classify the error, repair the trust contract, or stop promotion.
• Require at least one full, non-resumed handshake to accept a certificate or trust-store rotation. Test pooled connections and session resumption separately.
• Log socket reuse and TLS session reuse separately, without prompts, tokens, certificate bodies, private keys, or session/ticket data.

Define the route trust contract

Start with one row per route. Record the URL hostname, effective port, proxy or sidecar hop, expected server name, accepted CA source, and whether client authentication is required. Add the connection lifetime, session-cache owner, and the method used to obtain a full, non-resumed handshake for rotation acceptance. A route is ready only when the same row is satisfied by the client that will carry user traffic. A probe from a different host, resolver, proxy, or container image can verify the wrong trust environment.

Use the runtime’s secure client API instead of assembling verification from string checks. Python’s ssl documentation describes SSLContext as the object that manages settings and certificates. Its create_default_context() helper enables client-oriented secure defaults and can load system certificates; an explicit context can instead load a reviewed CA file with load_verify_locations. The important operational choice is to make that source visible and reproducible. Do not let one worker inherit a developer laptop bundle while another uses a minimal container image.

HTTPX lets a caller pass an SSLContext to a client, and its guide documents both a system-store approach and an explicitly configured CA bundle. This is useful for a private enterprise CA, but it also makes a bad deployment easy to hide: a context that loads the wrong file can reject the intended peer, while a context built with verification disabled can accept an impostor. Require a configuration check that states the trust-store path or source, verification mode, and bundle revision before the process becomes ready.

In Node, inspect the TLS client’s authorized state and authorizationError rather than treating a connected socket as authenticated. The Node.js TLS reference states that peer certificates are verified only during a full TLS handshake. When a previous session is resumed, verification is not repeated: authorized and authorizationError carry the result stored with that session. Consequently, authorized=true on a resumed session does not prove that the replacement certificate chain was checked against the current trust store. Keep rejectUnauthorized enabled and retain the default hostname check.

A custom trust source changes the set of certificates the process can use. curl’s documentation distinguishes native stores from file-based stores and warns that --insecure should not be used in production. HTTPX documents SSL_CERT_FILE, SSL_CERT_DIR, and explicit contexts. Treat environment variables, base-image updates, and sidecar mounts as configuration inputs. Capture a bundle or image revision in readiness evidence so a later worker can reproduce the decision.

Preserve identity while changing the connection

Failover systems sometimes dial a resolved address, a regional ingress, or a proxy address. The address used by the socket is a routing detail. The certificate identity is the hostname that the service contract names. Preserve that logical hostname for TLS server-name indication and certificate matching, and verify that the connector does not perform an unchecked second DNS lookup. If the proxy terminates TLS, document that separate trust boundary and validate the next hop according to its own contract.

Python’s client example pairs socket.create_connection() with SSLContext.wrap_socket(..., server_hostname=hostname), which makes the split explicit: connection routing and certificate identity are related but separate values. Node exposes the SNI name associated with a TLS socket and a checkServerIdentity hook. Use the client’s supported identity mechanism rather than comparing an address string with a certificate name. Keep the logical host in the TLS handshake and let the approved connector control the peer address.

Do not replace hostname verification with a certificate fingerprint copied from one incident. A leaf certificate rotates, a provider may use several valid intermediates, and a fingerprint does not express whether the connection reached the intended hostname. If a product requires certificate pinning, treat it as an additional reviewed control with a rotation plan, telemetry, and a rollback path. It is not a substitute for ordinary chain and hostname verification.

Client certificates are a separate concern. A private key and client certificate prove who the caller is to a server that requests client authentication; they do not prove that the server is the expected API. Keep client material in the secret manager, load it into the intended context, and retain server verification. Never place either certificate chain or private-key bytes in logs.

Require a full handshake for rotation acceptance

Separate three observations. connection_reused says whether the request used an existing socket. session_reused says whether a newly established connection resumed earlier TLS session state. A full handshake performs the certificate verification needed to validate a replacement chain. The Node reference documents tlsSocket.isSessionReused() for the second observation. A new TCP connection with connection_reused=false can still have session_reused=true.

For a Node rotation probe, create a new tls.createSecureContext() with the candidate CA configuration and call tls.connect() directly, outside the shared HTTP agent or connection pool. Supply the intended servername, keep rejectUnauthorized: true, retain the default server-identity check, omit the session option, and do not import saved session or ticket data. Node documents that callers can pass saved session data through that option to resume a connection. A separate probe context without that state avoids carrying the old session into the acceptance check.

After secureConnect, require socket.isSessionReused() === false and socket.authorized === true, with hostname verification enabled. Confirm that the controlled route presents the intended candidate certificate chain. Record a full-handshake result and the candidate trust-store revision, then close the probe and discard any session data it produces. A resumed result, an unknown reuse state, or unavailable handshake evidence cannot pass rotation acceptance. Keep the ordinary pooled-client and resumption checks as separate compatibility tests.

Run the probe from the production routing cell with the same hostname, proxy path, TLS policy, and candidate trust source as the route under test. Where a sidecar owns upstream TLS, the full-handshake evidence must come from that sidecar’s upstream hop; an application-side check cannot stand in for it. This isolated check validates the rotation path without requiring the fleet to disable useful session resumption.

Happy-path operator workflow

1. Identify the route generation and the exact client or sidecar that will make the request.
2. Load the reviewed CA source and client certificate, if required, into an isolated probe context without retained TLS session state. Record the candidate source revision and verification mode.
3. Resolve or select the route according to the approved network policy, then establish a new connection with the intended server name. For rotation acceptance, require a full handshake and confirm that session reuse is false.
4. Require successful chain validation and hostname identity against the intended candidate chain before sending an API request. A full handshake is not a model or response-contract check, so follow it with the smallest safe endpoint readiness request.
5. Record the verification result, handshake type, TLS version, route generation, socket reuse, session reuse, and sanitized error class. Do not record certificate bodies, authorization headers, prompts, generated text, or session/ticket data.
6. Promote fallback traffic only when endpoint readiness, remaining request deadline, capacity policy, and TLS trust all pass. For deadline arithmetic, see Design end-to-end deadline propagation for LLM API failover chains .

A useful event shape is:

{
  "event": "tls_route_check",
  "route_generation": 17,
  "route_id": "fallback-a",
  "connection_reused": false,
  "session_reused": false,
  "handshake_type": "full",
  "trust_store_revision": "ca-2026-10-01",
  "server_name_class": "configured-host",
  "verification": "passed_current_handshake",
  "rotation_acceptance": "passed",
  "endpoint_readiness": "passed",
  "error_class": null
}

This is an illustrative acceptance event, not an observed test result. A rotation check passes only when both reuse fields are false, the handshake is full, and verification succeeds against the candidate chain and trust-store revision. A resumed session can support continuity testing, but its retained authorization is not fresh rotation evidence. Hash or classify names and addresses when raw values are unnecessary. Keep the route generation so a delayed event from an old certificate, session, or pool cannot be mistaken for current evidence.

Error-path operator workflow

Stop rotation promotion when the full-handshake probe fails verification, resumes a session, or cannot establish whether resumption occurred. A resumed success is inconclusive for rotation acceptance; repeat the isolated probe without retained session state and keep the gate closed until a full handshake passes. Classify the failure before retrying: trust anchor missing, expired or not-yet-valid certificate, hostname mismatch, incomplete chain, client-certificate rejection, protocol mismatch, or peer-identity mismatch after a proxy hop. A retry through the same broken context will not repair the cause, and a retry with verification disabled creates a larger incident.

Compare the primary and fallback contexts without printing secret material. Check the effective hostname and SNI, CA source and revision, system clock, proxy termination point, client certificate selection, and the peer error code. If the primary succeeds and the fallback fails, do not infer that the provider is unavailable; the difference may be a container image, stale CA bundle, sidecar policy, or route-specific certificate.

When a CA bundle is intentionally rotated, stage the new bundle and test it against the current and replacement chains using full, non-resumed handshakes. Require the expected chain and hostname to pass and a missing-CA or hostname-mismatch fixture to fail. Then roll workers in a bounded wave. Keep the old trust anchor only for the overlap that the certificate plan requires, then remove it through the same reviewed configuration path. If the service presents an unexpected chain, stop and obtain an authoritative certificate and endpoint review. Do not paste a leaf certificate into a production image during an incident.

If the route uses a pooled client, keep in-flight requests bounded by the application deadline and perform the isolated full-handshake check after the trust-store change. Existing sockets can hide a broken cold path, and new sockets can resume cached TLS sessions without repeating certificate verification. Replacing the socket alone is insufficient. Validate the candidate chain with session reuse confirmed false, then test ordinary pooling and resumption separately. Destroying every connection at once can create a handshake surge. Coordinate pool turnover with the admission limit and the endpoint health and routing readiness check .

Keep TLS failures in the route decision record even if another provider is eligible. The gateway can use a separately reviewed alternate route with its own valid trust contract, but it should preserve the original failure class and action-level budget. The provider error classification guide gives the broader rule: classify before retrying, and do not let an SDK or fallback reset the user deadline.

Contract details to verify

OpenSSL’s verification API documents modes, callbacks, and verification depth. That is a reminder to keep library-level settings in the route contract rather than relying on an opaque default that may change with a base image or runtime upgrade. Verification depth is not a license to accept an arbitrary chain: it should match the certificate hierarchy the service has reviewed.

The trust contract should also state where TLS terminates. curl’s documentation treats HTTPS-to-proxy and proxy-to-server verification as separate connections with separate CA controls. If a sidecar terminates the outer connection and creates another upstream connection, the application’s green outer handshake does not authenticate the inner hop. Assign each hop an owner, a hostname, a CA source, and a failure code.

Failure modes

The fallback uses verify=False or rejectUnauthorized: false. This turns a route outage into an authentication gap. Restore verification, capture the error, and repair the trust source or endpoint identity.

The client dials an IP and checks the IP. The service certificate may be valid for the intended hostname but not the address. Keep the logical server name for SNI and hostname matching while controlling the dial target separately.

A custom CA file is copied into only some workers. The fleet then has split trust behavior. Roll out the bundle and its revision as one configuration unit and include it in readiness evidence.

A new socket resumes old TLS authentication. connection_reused=false does not imply a full handshake. Node documents that resumed sessions retain the prior verification result. Use an isolated probe without retained session state and require isSessionReused() to be false before accepting the replacement chain.

A reused socket hides a broken rotation. Existing connections can continue until their lifetime or idle limit. Run a bounded full, non-resumed handshake check after every trust-store or certificate change, then test pooled behavior separately.

A client certificate is mistaken for server trust. mTLS credentials authenticate the caller; they do not validate the API server. Keep both directions explicit.

A proxy changes the verification boundary. Document whether TLS ends at the proxy or upstream and validate each hop with the correct CA and hostname. Do not assume the outer handshake covers the inner connection.

Certificate details leak into diagnostics. Full chains, private keys, authorization headers, and URLs can be sensitive. Keep stable classes, revisions, and timing while redacting material.

Clock skew creates false certificate failures. Certificate validity depends on time. Check synchronized clocks and record only a coarse clock-health result; do not “fix” the error by weakening verification.

A trust context is mutated after use. Python documents that an SSLContext is designed to be shared but has limited mutation once used by a connection, and that changing verification settings or mTLS certificates can have surprising effects. Build a new context for a trust-store revision and roll clients in a bounded wave instead of mutating a live context in place.

FAQ

Does a successful TCP connection prove the fallback is safe?

No. TCP proves that a peer accepted a connection. TLS must still authenticate the peer, and the application must still validate the response contract.

Can we use the primary route’s CA bundle for every fallback?

Only when the routes intentionally share the same trust anchor and that choice is documented. A private endpoint or separate enterprise proxy may require a different reviewed CA source. Keep the source explicit rather than assuming that one bundle fits every route.

Is hostname verification enough without chain verification?

No. The name can match a certificate that is not anchored in a trusted CA. Require both chain validation and hostname identity.

Should a certificate error trigger another provider?

Usually it should stop the current route and enter a bounded operational decision. If a separately reviewed fallback has its own valid trust contract, the gateway may use it, but it must not replay blindly or hide the TLS failure.

How should we test a certificate rotation?

Use a disposable endpoint or staging route with known old and new chains. Require a full, non-resumed handshake against each intended chain using the candidate trust store, plus a denied hostname-mismatch or missing-CA fixture. Test pooled sockets and resumed sessions separately. Record both reuse states, verification outcomes, and bundle revisions; a resumed authorized=true cannot pass the rotation gate.

What should an incident packet contain?

Keep route generation, client/runtime version, trust-store revision, hostname class, connection reuse, session reuse, handshake type, TLS error class, endpoint-readiness result, retry count, and the terminal promotion decision. Exclude keys, private keys, certificate bodies, resumable session or ticket data, prompts, response bodies, and raw authorization headers.

Reader next step

Choose one fallback route and write its trust contract before the next release. Create an isolated probe using the candidate CA source without retained session state. Require a full handshake against the intended replacement chain with connection_reused=false, session_reused=false, successful authorization, and hostname verification enabled. Separately test the normal pooled path and session resumption so neither is mistaken for fresh certificate validation.

Deliberately test a hostname mismatch or missing-CA fixture in a disposable environment and confirm that the client stops before sending user data. Rotate the reviewed bundle through one canary worker and collect the full-handshake acceptance event from the component that owns upstream TLS. Expand only when that event, endpoint readiness, and deadline checks all pass. If session reuse is true or unknown, keep rotation acceptance pending.

If the route cannot show its trust store, server name, verification mode, and full-handshake rotation evidence, leave the candidate configuration out of automatic failover until that evidence exists. Pair the result with the site’s fallback decision log fields so an operator can reconstruct why a route was promoted or withheld.

Sources checked

• curl SSL CA Certificates — accessed 2026-10-16; documents CA-store selection, default certificate verification, server-name checks, and separate proxy verification.
• HTTPX SSL — accessed 2026-10-16; documents default HTTPS verification, CA bundles, explicit SSLContext configuration, client certificates, and the insecure verify=False option.
• Python ssl module — accessed 2026-10-16; documents secure default contexts, verification locations, limited post-use context mutation, and hostname-aware socket wrapping.
• Node.js TLS — accessed and rechecked 2026-10-16; documents client authorization state, certificate errors, CA settings, SNI, session resumption, isSessionReused(), and the fact that resumed sessions retain earlier verification results instead of repeating certificate verification.
• OpenSSL SSL_CTX_set_verify — accessed 2026-10-16; documents peer-verification modes, callbacks, and verification-depth controls.

These official references support the client and library behavior described here. They do not establish any provider’s current certificate chain, private endpoint, mTLS requirement, uptime, or failover guarantee; verify those deployment-specific facts separately.