Last reviewed: 2026-08-13

Direct answer

Keep one gateway-owned, provider-neutral transcript as the source of truth for every conversation. Store provider continuation identifiers only as route-specific metadata. When the primary route fails, rebuild the fallback request from the last committed transcript version instead of trying to transfer a provider’s opaque conversation identifier.

This separation matters because providers expose different state contracts. OpenAI’s conversation-state guidance describes independent, stateless requests alongside manual history replay, durable conversation objects, and response chaining. Anthropic’s Messages API guidance says the Messages API is stateless and that clients send the full conversational history. Google’s Gemini text-generation guidance documents server-managed multi-turn state through a previous-interaction identifier.

Those mechanisms are useful within their respective routes, but none should become the gateway’s only record. A canonical transcript gives the gateway a stable failover boundary: ordered user and assistant content, effective instructions, tool requests and results, attachment references, completion state, and a monotonically increasing version. Each provider adapter projects that record into its own request format.

Only committed items belong in the replayable transcript. Partial streamed text, an unresolved tool request, or a response whose outcome is unknown must remain provisional. On failover, resume from the last durable boundary, reconcile any external side effects, render a fresh provider request, and commit the fallback response exactly once.

Who this is for

This pattern is for engineers who operate multi-provider chat, assistant, or agent gateways and need a conversation to remain coherent when a model route times out, rejects a request, becomes overloaded, or loses its server-side state. It is especially relevant when conversations include streaming, tools, files, summaries, or concurrent client turns.

It is less useful for isolated, single-turn generation where the complete request already contains all required context. Even there, the same commit and attempt records can make ambiguous failures easier to diagnose.

Key takeaways

  • Make the gateway’s canonical transcript authoritative. Treat provider-side chains as accelerators, not the sole record.
  • Assign every transcript mutation a sequence number and version so concurrent turns cannot silently reorder history.
  • Separate committed conversation items from provisional stream chunks and incomplete tool activity.
  • Render each provider request from the canonical record through a versioned adapter; never translate one provider’s request payload directly into another provider’s payload.
  • Keep tool execution state outside free-form assistant prose and reconcile it before retrying or failing over.
  • Define deterministic context reduction before an emergency. Silent truncation during an incident can change the meaning of the conversation.
  • Log routing and state transitions with references, counts, and digests rather than raw prompts, responses, files, or personal data.

Sources checked

  • OpenAI: Conversation state explains manual history management, durable conversation state, and response chaining. It supports the distinction between application-managed history and provider-managed continuation.
  • Anthropic: Using the Messages API explicitly describes the API as stateless and requires the full conversation history for multiple turns. It also shows that earlier assistant turns can be supplied in the history.
  • Google AI for Developers: Text generation describes multi-turn interactions chained with a previous-interaction identifier and server-side conversation-history management.

Together, these public contracts establish the portability problem: conversation state can be represented by full client-managed history or by provider-managed chains. A reliable cross-provider gateway therefore needs its own representation that can generate either form.

Contract details to verify

Define the canonical record

Use an append-oriented record with an explicit schema version. Each item should include a conversation reference, item sequence, turn sequence, semantic kind, lifecycle state, creation time, and content-block list. Semantic kinds can include instruction, user message, assistant message, tool request, tool result, and attachment reference. Keep route-specific response identifiers in a separate metadata map so they cannot be mistaken for portable conversation content.

Preserve content as typed blocks rather than flattening everything into one string. Text, file references, tool arguments, tool results, and refusal or completion states have different replay rules. Record the instruction scope as data as well: an instruction that applies to the whole session is not interchangeable with one introduced halfway through a conversation.

A committed item must be immutable. Corrections should append a superseding item or create a new transcript version, not rewrite history in place. This makes a failover attempt reproducible and lets an operator identify the exact state used for a response.

Build provider projections

Each adapter should accept a canonical transcript version and return a provider request plus a projection report. The report should name omitted items, transformed roles, attachment substitutions, context reductions, and unsupported features. Reject a projection when a required instruction, tool result, or user attachment cannot be represented safely.

For a route that accepts full history, render the ordered committed items directly into the supported role and content structure. For a route that supports server-managed continuation, the adapter may use its saved continuation identifier only when it belongs to the same route and transcript version. If that identifier is missing, stale, or rejected, rebuild from canonical state rather than starting an empty conversation.

Test adapters with transcripts containing alternating turns, mid-conversation instructions, tool results, refusals, partial attempts, and attachments. The expected result is semantic preservation, not byte-for-byte request equality.

Happy-path operator workflow

  1. Read the latest committed transcript and its version.
  2. Validate and append the new user turn using an optimistic version check.
  3. Select a healthy route and project that exact transcript version into the provider contract.
  4. Record an attempt with route, projection version, start time, and transcript version before dispatch.
  5. Keep streamed output provisional. It may be shown to the client, but it is not yet replayable history.
  6. When a terminal response arrives, normalize its visible content and completion state.
  7. Commit the assistant item once, advance the transcript version, and mark the attempt successful.
  8. Save any provider continuation identifier against the route and committed version. Return the committed result to the client.

Error-path operator workflow

  1. When a timeout, disconnect, or malformed terminal response occurs, mark the attempt outcome as ambiguous or failed. Do not promote buffered stream fragments into a completed assistant turn.
  2. Freeze the transcript version used by the failed attempt. A concurrent user turn must either wait or start from a later, separately versioned state.
  3. Consult the tool-action ledger. If an external action started or completed, reconcile its durable result before making another model call.
  4. Apply the retry and failover policy. Select a fallback only if the remaining time, attempt, cost, and safety budgets allow it.
  5. Project the same committed transcript version to the fallback. Do not include the failed route’s provisional prose as if the user had received a complete answer.
  6. If the fallback cannot accept the full context, run the approved compaction or reject the failover with a controlled degradation response. Never drop early instructions silently.
  7. Dispatch the fallback, commit one terminal assistant result, and link both attempts to the same user action.
  8. Preserve the failure classification and state boundary for later review.

Log sanitized state transitions

A useful event records enough control-plane evidence to reconstruct the decision without copying conversation content:

{
  "event": "llm_failover",
  "conversation_ref": "conv_42",
  "turn_seq": 18,
  "transcript_version": 41,
  "transcript_digest": "8fa2c1",
  "attempt_ref": "try_17",
  "from_route": "primary-a",
  "to_route": "fallback-b",
  "failure_class": "timeout",
  "last_committed_item": 37,
  "stream_state": "partial_discarded",
  "tool_effect_state": "not_started",
  "projection_version": "v3",
  "context_action": "none",
  "outcome": "resumed"
}

Also record timestamps, latency buckets, item counts, content-type counts, context-size estimates, and the policy rule that authorized failover. Keep raw messages, tool payloads, file contents, personal data, and authentication material out of ordinary reliability logs. If deeper incident evidence is required, place it behind a separately governed access and retention path.

Failure modes

A provider identifier becomes the only history. The fallback cannot resolve an identifier created by another service, so it receives only the latest user message. The answer may look fluent while ignoring prior constraints. Prevent this by retaining enough canonical content to rebuild every active conversation.

Concurrent turns reorder the transcript. Two workers read version 40 and both append version 41. One assistant then answers the wrong latest turn. Use compare-and-swap or a per-conversation sequencer, and reject stale commits.

Partial output is replayed as a completed answer. A stream disconnects after several sentences, and the gateway appends those sentences with a normal assistant role. The fallback treats the fragment as established history. Store stream chunks under the attempt, then promote them only after a valid terminal outcome.

A tool action runs twice. The first model call triggers an external action, but the transport fails before the gateway receives the final response. Replaying the transcript can trigger the same action again. Pair the canonical transcript with durable action identifiers and follow the site’s duplicate tool-side-effect controls before dispatching a fallback.

Instruction authority changes during projection. A session-wide instruction is rendered as ordinary conversation text, or a later instruction is moved to the beginning. The fallback sees the same words with different precedence. Make instruction scope explicit and fail adapter conformance tests when placement changes.

Context reduction drops the wrong history. A fallback has a smaller usable context and the emergency path removes early requirements while preserving casual recent turns. Define priority, summary provenance, and validation in advance. The related context-window admission controls can prevent an oversized transcript from reaching the failover stage unexamined.

Attachments are present only in provider storage. The canonical record contains a route-specific file reference that the fallback cannot retrieve. Store a governed, provider-neutral attachment reference and let each adapter create or resolve its own provider copy before the conversation depends on it.

A stale continuation points to an older version. The provider chain was last updated at version 35, but the canonical record is at version 41. Continuing from the stale chain loses six items. Bind every continuation identifier to its transcript version and rebuild when the versions differ.

FAQ

Can the gateway pass a conversation identifier directly to another provider?

No. Treat a provider continuation identifier as metadata understood by that provider’s adapter. Cross-provider failover should start from the canonical transcript and produce a new request under the fallback contract.

Should partial streamed text be included when failing over?

Not as a completed assistant message. Keep it attached to the failed attempt. If the product displayed the fragment, record that delivery state separately so the fallback can avoid confusing repetition, but do not mislabel an incomplete generation as settled conversation history.

Should the canonical transcript contain hidden reasoning?

Store only contractually available items that the application is permitted and required to retain. Do not invent, translate, or depend on hidden reasoning. If a provider returns opaque continuation data needed only for that route, keep it in restricted adapter metadata rather than projecting it to another provider.

How should summaries be handled?

Treat a summary as a versioned transformation with source boundaries, a generation policy, and validation status. Retain the relationship to the items it replaces. A fallback should never receive an unlabeled summary that could be mistaken for a user statement or original instruction.

What happens when traffic returns to the primary provider?

Project the current canonical transcript back to the primary route. Do not try to reverse-translate the fallback provider’s payload or continue a primary chain that predates the latest committed version. Create a new primary continuation when necessary and bind it to the current transcript version.

Reader next step

Choose one representative multi-turn workflow and export its state into a provider-neutral fixture. Include an instruction, several user and assistant turns, one tool result, and one provisional failed attempt. Then implement projections for the primary and fallback routes and run two drills: a disconnect during streaming and a timeout after a tool action.

The drill passes only when both routes receive the same committed facts and effective instructions, the external action occurs at most once, provisional text stays out of durable history, and the final assistant response commits once against the expected transcript version. Turn those assertions into adapter conformance tests and run them whenever a provider contract, model route, tool schema, or compaction policy changes.