Last reviewed: 2026-08-05

Direct answer

LLM fallback shadow testing sends a controlled copy of an eligible production request to a standby route while the approved primary route remains the only source of the user-facing response. The shadow result is evidence, not an emergency substitute. It can be measured or retained for an approved comparison, but it must not reach the caller, update application state, populate a production cache, or execute a proposed tool action.

That design follows the central pattern in Istio’s traffic-mirroring guide : a copy of live traffic goes to a mirrored service outside the primary request path, and operators can mirror only a portion of traffic. The Amazon SageMaker shadow-test documentation makes the response boundary explicit: only the production response returns to the application, while the shadow response may be discarded or logged for offline comparison.

For an LLM gateway, build the mirror after request eligibility and policy checks. Copy the normalized model input and the contract-relevant settings, not the entire inbound request context. Give the shadow call its own timeout, concurrency ceiling, and kill switch. If the candidate proposes a tool call, capture the proposal only in a permitted, sanitized form; do not execute it. If those boundaries cannot be guaranteed for a workload class, keep that class out of live shadow traffic and test it with an approved offline prompt set instead.

Happy-path operator workflow

  1. Write an experiment card naming the primary route, shadow route, model references, eligible request classes, sample rule, duration, resource ceiling, evaluator version, promotion gates, and stop conditions.
  2. Confirm that the candidate can accept the same logical request contract. Check prompt-template version, role ordering, output limits, streaming mode, structured-output requirements, and any tool definitions.
  3. Select a small, policy-approved sample. Attach a comparison identifier that is not derived from user content, then dispatch the primary and shadow requests independently.
  4. Return the primary result without waiting for the shadow. Keep the candidate outside every active user-response and failover decision.
  5. Record paired operational outcomes such as status class and latency, followed by deterministic contract checks and an approved quality evaluation.
  6. Review results by request class, not only as one aggregate. Increase the sample only when the primary remains unaffected and the candidate continues to meet the predefined gates.
  7. End the shadow run at its stated expiry. Promote the candidate only through a separate, reviewed routing decision.

Error-path operator workflow

  1. If the shadow call times out or returns an error, classify that outcome, cancel any remaining shadow work, and leave the primary path unchanged.
  2. If a shadow response is malformed, fails a required policy check, or proposes a prohibited side effect, mark the comparison as failed and stop that request class from being mirrored.
  3. If the primary call fails, follow the currently approved production behavior. Do not return the experimental shadow result merely because it succeeded.
  4. If paired telemetry is missing or ambiguous, exclude the pair from promotion evidence rather than treating absence as success.
  5. If shadow traffic raises shared-resource pressure or affects primary latency, reduce the sample to zero with the kill switch, preserve sanitized evidence, and investigate before restarting.

Who this is for

This guide is for engineers who own LLM gateways, standby model routes, incident failover, model migrations, or production evaluation. It is especially useful when synthetic probes prove that an endpoint is reachable but do not show how the candidate handles the request shapes that users actually send.

The workflow assumes operators can separate user-visible execution from observation. Teams that cannot suppress tool execution, writes, notifications, memory updates, or cache effects should begin with an offline evaluation dataset. Shadowing is a promotion test for a candidate route; it is not permission to duplicate every workload.

Key takeaways

  • The primary response remains authoritative throughout a shadow test.
  • Mirror only eligible, policy-approved request classes and use a bounded sample.
  • Isolate shadow time, capacity, storage, cache behavior, and side effects from production.
  • Compare transport health, latency, output contracts, and task quality separately.
  • Treat missing pairs and evaluator failures as unknown, not as passes.
  • Define promotion and stop conditions before collecting results.
  • A successful shadow run supports a later routing review; it does not activate failover automatically.

Sources checked

  • Istio: Mirroring documents copying live traffic to a mirrored service outside the critical primary path and selecting a mirror percentage. This supports the routing topology and bounded-sampling pattern.
  • Amazon SageMaker: Shadow tests documents routing real-time inference copies to a shadow variant, returning only production responses, and using latency, errors, and offline response comparison before promotion.
  • Microsoft Engineering Fundamentals: Shadow Testing describes running current and candidate environments side by side with replicated production traffic, withholding the candidate response from users, and comparing the collected results.
  • Amazon Bedrock: Create a model evaluation job using built-in metrics documents prompt-dataset evaluation, model-as-judge scoring, and evaluation of supplied inference responses. This supports an LLM-specific offline comparison stage after paired shadow responses are captured.

Together, these sources support the architecture of out-of-band mirroring, user-response isolation, realistic traffic comparison, operational measurement, and structured model-output evaluation. They do not supply a universal sample percentage or promotion score. Those thresholds must come from the application’s own reliability and quality contract.

Contract details to verify

Start with invariants that remain true even when the candidate misbehaves. The user receives only the approved production response. The primary call never waits for shadow completion. A shadow failure cannot trigger another user-facing retry. The shadow executor cannot commit a tool action or write to production state. Operators can stop mirroring without redeploying the primary path.

Next, verify request parity. The two routes should receive the same logical user task after approved sanitization, but each provider adapter may serialize that task differently. Record the prompt-template version, route configuration version, model reference, sampling settings, output cap, streaming flag, structured-output schema, and tool-schema version. A candidate that receives a different instruction stack is not being compared fairly. The related response-contract review can help define deterministic shape checks before quality scoring begins.

Define the side-effect boundary explicitly. Tool-capable requests may be mirrored only when the shadow executor records proposed calls without dispatching them, or when every destination is a purpose-built test double. Do not silently remove tool definitions if tool compatibility is part of the fallback contract; preserve the definitions and block execution instead. The tool-call fallback contract guide covers the adjacent compatibility checks.

Build comparison gates in layers:

  1. Routing gate: Was the request admitted to the intended candidate route with the expected configuration?
  2. Operational gate: Did the candidate complete within its shadow deadline, and what status and latency did it produce?
  3. Contract gate: Did the response parse, satisfy the required schema, finish in an allowed state, and keep tool proposals within the declared schema?
  4. Policy gate: Did the result satisfy the application’s existing safety and data-handling rules?
  5. Quality gate: Did an approved rubric score the response for the task, and did a human sample review the cases that matter most?
  6. Promotion gate: Did the candidate pass across representative request classes for the declared test window without harming the primary path?

Keep logs useful without turning them into a second content store. A sanitized comparison event can look like this:

{
  "event": "llm_shadow_pair",
  "comparison_id": "cmp-042",
  "workload_class": "support",
  "sample_rule": "v3",
  "primary_route": "route-a",
  "shadow_route": "route-b",
  "primary_status": 200,
  "shadow_status": 200,
  "primary_latency_ms": 840,
  "shadow_latency_ms": 910,
  "schema_match": true,
  "tool_mode": "blocked",
  "content_capture": "[REDACTED]",
  "evaluation_state": "queued"
}

Useful fields include event time, comparison identifier, request class, sampling-rule version, primary and shadow route references, model references, prompt-template version, streaming mode, input-size bucket, status class, normalized error class, latency, completion state, schema-validation result, proposed tool name category, side-effect-guard outcome, evaluator version, rubric result, and retention class. Avoid raw prompts, raw outputs, direct user identifiers, or complete request headers in general operational logs. If an approved quality review requires content, store the minimum permitted material in a separately controlled evaluation dataset and reference it with an opaque comparison identifier.

Finally, define how paired records close. A completed primary response plus a shadow timeout is a valid pair with a failed shadow outcome. A missing correlation record is not. A failed evaluator produces an unknown quality result, not a passing one. These distinctions keep promotion evidence reviewable.

Failure modes

The shadow answer reaches the caller. A route can be labelled shadow while still being wired into an error handler. Test the invariant directly: primary failure must not make the experimental response eligible for return.

A proposed action becomes a real action. Mirrored tool requests can duplicate messages, writes, or other external changes if the normal executor consumes them. Use a record-only executor or exclude the workload class.

Shadow work competes with the primary. Out-of-band routing removes the candidate from the primary response chain, but shared capacity can still couple the two paths. Give the experiment explicit concurrency, timeout, and resource ceilings, then stop it when primary health changes.

The sample is convenient but unrepresentative. Mirroring only short text requests can hide failures in long inputs, streaming, structured output, multilingual requests, or tool-capable tasks. Report results by declared request class and list excluded classes.

The two routes receive different tasks. Divergent prompt versions, missing system instructions, different output limits, or provider-specific transforms can create a misleading comparison. Log configuration versions and inspect representative serialized requests in an approved test environment.

Exact-text comparison creates false alarms. Define contract checks and task rubrics instead of requiring identical wording. Keep deterministic failures separate from quality judgments.

A judge score becomes the only promotion gate. Model-as-judge evaluation can organize response review, but the operator still needs operational checks, deterministic validators, representative coverage, and human examination of consequential cases.

Telemetry cannot pair the outcomes. If primary and shadow records use different identifiers or clocks without a reliable join, apparent regressions may be comparison errors. Reject unpaired evidence.

The evidence store captures too much. Logging complete prompts and responses can expand the data exposed by the test. Default operational logs to metadata and keep any permitted content review separate, minimal, and time-bounded.

A small shadow sample is mistaken for outage capacity proof. A candidate can look healthy at sampled volume and still be unproven for full failover. Treat shadow evidence as one gate, then test capacity and failure transitions separately.

FAQ

Is shadow testing the same as sending a canary response to a few users?

No. In the pattern supported by the refetched sources, the production response remains the only response returned to the application. User exposure belongs to a later, separately approved rollout stage.

Should every request be mirrored?

Not by default. Start with explicitly eligible request classes and a bounded sample that fits the candidate’s resource and data-handling contract. Expand only after reviewing primary impact and candidate results.

What happens when the primary fails but the shadow succeeds?

The shadow success is useful evidence, but it must remain invisible during the experiment. The user request follows the currently approved production failure behavior. Otherwise, the test has silently become an unreviewed failover path.

Can a string diff determine whether the fallback is good enough?

Usually it should not be the sole test. Use exact checks for schema and other deterministic requirements, then use a task-specific rubric for response quality. When a model judge is used, record its version and retain human review for important or ambiguous cases.

Can tool-capable requests be included?

Yes, but only when proposed tool calls can be observed without execution, or when every destination is an isolated test double. If the gateway cannot enforce that boundary, exclude those requests from live shadowing.

Does a passing shadow run prove the route is ready for a full outage?

No. It provides evidence about sampled real request shapes and candidate behavior. Full promotion still needs contract review, capacity evidence, failure-path testing, and an explicit routing decision.

Reader next step

Create a one-page shadow-test card for one low-risk request class. Name the two route versions, eligible sample, side-effect guard, resource ceiling, sanitized fields, quality rubric, stop conditions, owner, and expiry. Run the smallest useful sample, inspect paired failures first, and record excluded traffic classes so the result cannot be overstated.

Before promoting the route, complete the controlled fault-injection exercise and review the candidate against the fallback evidence checklist . The next decision should be explicit: reject the candidate, repair and repeat the shadow run, or advance it to a separately controlled rollout.