Last reviewed: 2026-08-31

Direct answer

Prevent split-brain routing by treating a CometAPI fallback policy as one immutable, versioned release rather than a set of independently changing flags. The policy revision should bind the ordered routes, model references, request contract, required capabilities, fallback error classifier, attempt limits, timeouts, and optional official-provider switch. Every gateway replica should report three separate states: the revision it was asked to load, the revision it successfully loaded, and the revision currently making routing decisions.

A control-plane write is not proof of convergence. Stage the new bundle without activating it, validate every referenced route, canary it in an explicitly isolated cohort, and require acknowledgements for the exact revision and digest. Promote only when the eligible cohort has loaded the same content and its probes are healthy. At activation, switch policy at a request boundary and keep each in-flight user action pinned to the revision on which it began.

A distributed switch is rarely instantaneous, so traffic eligibility is the final guardrail. During a normal cutover, a replica whose active revision differs from the production revision should not remain in the shared serving pool. During a deliberate canary, two revisions may serve at once only when traffic is intentionally partitioned into named cohorts and metrics retain the revision dimension. If a replica rejects the policy, misses the convergence deadline, or reports a different digest, keep the last proven revision active and remove the divergent replica from new traffic.

Who this is for

This guide is for platform engineers, gateway owners, and on-call operators running multiple LLM routing replicas. It is especially relevant when fallback behavior is distributed through Kubernetes configuration, an xDS-style control plane, feature controls, or a custom configuration service.

The focus is not choosing the best model. It is making sure every production replica applies the same already-approved routing contract, while preserving a controlled canary and a fast return to the last proven revision.

Key takeaways

  • Version the complete fallback contract, not just the primary and secondary model names.
  • Track desired, loaded, and active revisions separately; equality among them is a serving invariant, not a deployment detail.
  • Validate current model, provider, capability, endpoint-family, and pricing assumptions before activation.
  • Keep fallback classification narrow. Transport failures, timeouts, 408, 429, and verified temporary 5xx conditions are different from invalid requests or unsupported parameters.
  • Treat asynchronous configuration propagation as untrusted until each serving replica reports the expected active digest.
  • Roll back by repointing activation to an intact, previously proven bundle rather than editing the failed bundle in place.
  • Preserve revision-aware, sanitized decision logs so operators can distinguish provider trouble from policy divergence.

Sources checked

  • CometAPI’s model and provider fallback guide recommends the order CometAPI primary model, CometAPI fallback model, then an optional matching official provider. It also separates fallback-worthy connection errors, timeouts, 408, 429, and temporary 5xx responses from invalid requests, authentication failures, and unsupported parameters.
  • CometAPI model catalog documentation describes a public catalog containing model IDs, provider metadata, feature tags, endpoint hints, context and completion limits when available, and public pricing fields. These are useful preflight inputs, but an absent field should not be treated as proof of a capability.
  • Envoy xDS protocol documentation explains resource versions, acknowledgements and rejections, last-valid-configuration retention, and the eventual-consistency behavior of separate resource streams. It also describes aggregated transport for cases that need explicit sequencing.
  • Kubernetes ConfigMaps documentation states that projected volume updates are eventual, values consumed as environment variables require a pod restart, and a ConfigMap mounted through subPath does not receive updates. Those consumption modes can leave replicas on different policy revisions unless the rollout accounts for them.

Contract details to verify

Make one immutable policy unit

A policy revision should contain everything needed to reproduce a routing decision. The following values are illustrative rather than product defaults:

policy_revision: "fp-042"
policy_digest: "7d31ac09b2e4"
request_contract: "chat-v3"
routes:
  - route_id: "comet-primary"
    model_ref: "primary-model"
    capability_set: ["text", "streaming"]
  - route_id: "comet-secondary"
    model_ref: "fallback-model"
    capability_set: ["text", "streaming"]
fallback_on:
  - "connection-error"
  - "timeout"
  - "http-408"
  - "http-429"
  - "verified-temporary-5xx"
do_not_fallback_on:
  - "invalid-request"
  - "unsupported-parameter"
  - "authentication-failure"
per_attempt_timeout_ms: 4500
max_attempts: 2
official_provider_enabled: false

The digest should be calculated from the canonical policy content. A revision label without a digest is insufficient because two replicas could report the same label while holding different bytes. Store credentials outside this artifact, and do not include prompt content or customer data in it.

Preflight each route against the current CometAPI catalog. Confirm that the model reference exists, the expected provider is represented, the endpoint family matches the request format, and the required capability metadata is present. Review context, completion, and pricing fields when they are relevant and available. A matching model name alone does not establish request or response compatibility; the CometAPI fallback guide explicitly warns that switching model families can require request conversion, response normalization, and capability verification.

Validate the policy as a graph as well as a document. Every route reference must resolve, each failure class must have one outcome, total attempts must be bounded, and the sum of per-attempt timeouts must fit the user-action latency budget. If an official-provider route is enabled, verify its separate account, billing controls, client configuration, and operational readiness before promotion.

Happy path: stage, canary, and activate

  1. Record the current active revision, digest, route success rate, fallback rate, latency, and error-class distribution. Keep its immutable bundle available for rollback.
  2. Build the new bundle and run syntax, schema, referential-integrity, capability, request-shape, and response-shape checks. Exercise both success and error cases: a valid request should complete on the intended route, a verified timeout should advance exactly once, and an invalid request should stop without fallback.
  3. Distribute the bundle to a non-active slot. Each replica reports its desired revision, loaded revision, digest, load result, and validation time. A load error is a rejection, not an implicit acceptance of partial content.
  4. Activate the bundle in a named canary cohort. Keep that cohort separate from the stable pool and attach the policy revision to every metric and decision log. Run synthetic requests for every route and compare status classes, latency, output contract, fallback frequency, and spend indicators with the baseline.
  5. Promote the activation pointer only after the required cohort acknowledges the exact digest and the canary meets its gates. Gate production readiness on the active revision so a late or restarted replica cannot rejoin with stale policy.
  6. Switch at a request boundary. An in-flight action retains its starting revision for all attempts; new actions use the promoted revision. Finish the rollout only when every eligible production replica reports the promoted active revision and digest.

Error path: freeze, isolate, and recover

If a replica rejects the bundle, preserve the previous active revision and capture the rejection class. Correct the policy as a new revision; do not mutate the rejected artifact while retaining its revision label.

If an acknowledgement is missing, mark that replica ineligible for new traffic. Determine whether it is delayed, disconnected, consuming configuration through a mode that requires restart, or reading a mount that will not update. Do not lower the convergence threshold merely to finish the deployment.

If canary health regresses, stop promotion, remove the canary from normal traffic, and point it back to the last proven bundle. If regression appears after wider activation, move the production activation pointer back, drain replicas that still report the failed revision, and verify that the active-revision cardinality has returned to one in the stable pool.

When the control plane is unavailable, gateways should continue with their last proven complete policy rather than assemble a partial policy from whatever resources arrive. Before a recovered replica serves again, require it to load and activate the current production digest.

Log the decision without logging the customer

A useful event records policy and routing state, not raw prompts, response bodies, request headers, account data, or secrets. One sanitized shape is:

{
  "event": "fallback_policy_decision",
  "request_id_hash": "4fa2b981",
  "gateway_replica": "gw-07",
  "policy_desired": "fp-043",
  "policy_loaded": "fp-043",
  "policy_active": "fp-042",
  "policy_digest_prefix": "7d31ac09b2e4",
  "route_id": "comet-secondary",
  "attempt_index": 2,
  "decision": "hold-old-policy",
  "decision_reason": "revision-not-active",
  "failure_class": "policy-revision-mismatch",
  "http_status": null,
  "elapsed_ms": 18,
  "catalog_checked_at": "2026-08-31T00:00:00Z"
}

Also record the rollout cohort, activation transition time, policy load result, classifier version, request-contract identifier, and whether the replica was serving-eligible. Use bounded enums for failure classes. If an upstream diagnostic unexpectedly contains sensitive material, replace that value with [REDACTED] before storage.

Failure modes

  • A configuration write is mistaken for convergence. The store contains the new policy, but some replicas still route with the old one. Desired revision metrics look healthy while active revisions disagree.
  • ConfigMap consumption modes are mixed. A volume projection updates eventually, an environment-based value changes only after restart, and a subPath mount does not receive the update. A rolling restart can therefore create an unplanned mixed-revision window.
  • Only some clients accept a dynamic update. With xDS-style delivery, a rejecting client can retain its last valid configuration while peers accept the new version. Without revision-aware readiness, both behaviors remain behind the same load balancer.
  • Related resources arrive out of order. Independent xDS resource streams use an eventual-consistency model. A route, cluster, and endpoint dependency can temporarily represent different points in the rollout unless sequencing and warming are designed deliberately.
  • The classifier changes on only part of the fleet. One replica advances on an invalid request while another correctly stops. This both hides request defects and makes incident evidence appear provider-dependent.
  • A model reference exists but the route contract is incompatible. The fallback lacks a required capability, uses a different request family, returns a different response shape, or has an unverified limit. Catalog presence alone does not prove equivalence.
  • Attempt timeouts are reviewed individually. Two or three reasonable-looking timeouts can exceed the user-action budget when added together, especially after queueing and network delay.
  • An optional official route is enabled before it is operationally ready. The policy reaches a separate account or billing boundary that has not been validated, turning a primary-route incident into a second failure.
  • Canary data loses its revision label. Aggregated metrics blend stable and canary outcomes, so the new policy appears healthy because most traffic still uses the old one.
  • Rollback edits the failed bundle. Operators cannot prove which content a revision represented, and replicas may retain different cached forms. Rollback should select a preserved bundle with a known digest.

FAQ

What counts as split-brain fallback routing?

It is the condition in which replicas eligible for the same unpartitioned traffic use different active policy content. Different desired or staged revisions are acceptable; different active revisions in one stable serving pool are not.

Can two revisions be active during a canary?

Yes, when it is intentional. Give stable and canary traffic explicit cohort boundaries, keep the revision on every metric and log, and prevent random load balancing from moving one user action between policies. The canary becomes unsafe when its second revision is accidental or unobservable.

Does an acknowledgement prove the fallback works?

No. It proves that a particular client accepted a configuration response or that your loader accepted a bundle, depending on the delivery system. It does not prove route reachability, request compatibility, output shape, latency, or account readiness. Pair acknowledgements with route-specific synthetic checks and canary health gates.

Should every server error trigger the next route?

No. The CometAPI guidance supports temporary 5xx conditions, not an undifferentiated rule that advances on every failure. Maintain verified error classes, keep invalid requests and unsupported parameters terminal, and review unknown errors before adding them to the classifier.

Can a ConfigMap remain the source of truth?

It can store policy data, but its propagation behavior is not a fleet-wide activation transaction. Add an immutable revision and digest, expose observed state from each replica, account for restart requirements, and gate serving readiness on the active revision. If the application cannot report what it loaded, the operator cannot prove convergence.

What is the safest rollback target?

Use the most recent immutable revision that passed route probes and production gates. Move the activation pointer back, remove mismatched replicas from new traffic, and confirm the stable pool reports one active digest before declaring recovery.

Is an official-provider fallback required?

No. The CometAPI guide presents it as an optional final layer after a fallback model inside CometAPI. Keep that switch off unless the separate route has been tested and its account, billing, timeout, and monitoring controls are ready.

Reader next step

Start by adding desired, loaded, and active policy revision fields to every gateway replica, then alert when the stable serving pool reports more than one active digest. Package the current fallback chain into an immutable bundle, validate its model and capability assumptions, and rehearse both a successful canary and a rejected update.

Use the fallback decision logging guide to make each routing choice reviewable, and pair the rollout with model alias drift checks before promotion.

When your revision gates and rollback path are ready, Start with CometAPI .