Last reviewed: August 4, 2026
Direct answer
CometAPI fallback fault injection should test a boundary you control: a service double, controlled proxy, gateway adapter, or isolated test environment. Begin with a healthy no-fault control, inject one realistic failure into a tightly selected cohort, compare the observed user outcome with a written hypothesis, then remove the fault and verify recovery. Do not try to prove resilience by deliberately disrupting a public upstream service.
A route switch alone is not a passing result. The test must show that the complete user action stays within its deadline and attempt budget, produces no duplicate user-visible response, preserves required output and safety contracts, respects cancellation, remains inside its spend limit, and leaves enough sanitized evidence for an operator to explain what happened.
AWS resilience-testing guidance specifically recommends controlled simulations for external dependencies, including blocked access, latency, dropped communication, and DNS or connection failures. Before building the experiment, bind fallback behavior to one user action so the test has an unambiguous unit of success.
Who this is for
This guide is for platform engineers, SREs, and application teams that operate CometAPI-backed LLM features with a primary route and one or more fallback routes. It is most useful when the routing logic already exists but has only been exercised through unit tests, manual toggles, or real incidents.
The approach also fits teams preparing a deployment gate or game day. You should already know which application component owns routing, deadlines, cancellation, and response delivery. If ownership is unclear, settle that contract before injecting faults.
Key takeaways
Test the user outcome, not merely whether traffic reached a fallback route.
Keep the first experiment synthetic, narrowly scoped, reversible, and separate from unrelated traffic.
Run a healthy control through the same adapter before activating a fault.
Cover latency, connection, status, empty-response, malformed-response, and slow-chunk behavior rather than relying on a single server-error test.
Define abort conditions and cleanup before the run, then verify that both actually work.
Retain sanitized decision and timing evidence so a pass can become an automated regression test.
A concrete operator workflow
Prepare the experiment card. Name one primary route, one eligible fallback route, the synthetic cohort selector, the injected fault, and the maximum number of user actions. Write a falsifiable hypothesis such as: if the adapter receives a connection reset before any response is committed, it will make at most one eligible fallback attempt and return one valid result within the action deadline. Record separate abort conditions for customer impact, unexpected route expansion, attempt count, latency, and estimated spend. Use thresholds derived from your own service objectives; there is no universal safe value.
Run the happy path. Keep fault injection disabled and send the synthetic request through the exact adapter and policy configuration under test. Confirm that the primary route is selected, the response satisfies the application contract, only one attempt is recorded, and the cleanup control reports no active fault. This control catches broken fixtures and stale routing configuration before they are mistaken for resilience findings.
Run the error path. Enable one fault for the selected cohort. Start with a pre-response condition such as a connection reset, bounded delay, or chosen HTTP error. Verify the observed error category, route decision, attempt count, fallback eligibility check, final response classification, elapsed time, and estimated usage. Then test a post-header or slow-chunk condition separately; it has a different response-commit risk. WireMock’s fault-simulation documentation shows practical mechanisms for fixed and random delays, chunked delivery, empty responses, malformed chunks, random data followed by closure, and connection resets.
Abort and recover deliberately. Trigger the test abort in a safe rehearsal instead of assuming it works. Disable the injector, remove temporary selectors, restore the known-good routing configuration, and run the healthy control again. If the control does not recover, the experiment remains failed even if fallback worked during injection.
Review and promote. Compare the control and faulted runs against the same steady-state measures. A passing experiment should become a versioned regression test tied to routing, adapter, deadline, and response-contract changes. The Azure Chaos Studio overview provides a useful model: scope controls the blast radius, experiments have explicit steps and actions, completed runs produce evidence, and continuous validation can act as a deployment gate.
Keep a sanitized evidence record
Record operational metadata, not prompts or generated content. Useful fields include:
experiment ID, scenario version, environment, and cohort;
synthetic action ID and trace correlation ID;
route role, route identifier, model alias, and attempt index;
fault type, injection point, activation time, and removal time;
connect, first-byte, and completion timing where applicable;
normalized status or error category;
response-commit state and cancellation state;
fallback decision reason and policy version;
input and output usage counts when available to the application;
estimated spend guardrail result;
final user-outcome classification and cleanup result.
A compact event can look like this:
{
"event": "fallback_experiment_result",
"experiment_id": "fi-042",
"action_id": "act-017",
"cohort": "synthetic",
"fault_type": "connection_reset",
"route_role": "fallback",
"attempt_index": 2,
"response_committed": false,
"outcome": "recovered",
"content": "[REDACTED]"
}
Exclude request and response bodies, transport headers, cookies, and unfiltered query strings. Hashing sensitive content is not automatically safe if the input space is small or predictable; prefer a non-content correlation identifier created by your application.
Sources checked
AWS Well-Architected: Test resiliency using chaos engineering supports realistic external-dependency faults, explicit hypotheses, scoped runs, stop conditions, rollback, observability, and retaining experiment results.
Microsoft: What is Azure Chaos Studio? supports blast-radius scoping, structured experiment steps, incident reproduction, game days, reports, and continuous deployment validation.
Principles of Chaos Engineering supplies the steady-state and hypothesis model, identifies malformed responses and traffic spikes as realistic variables, and emphasizes minimizing experiment blast radius. It is foundational guidance with a stated last update in 2019.
WireMock: Simulating Faults supplies implementation examples for error status responses, long-tail delays, chunked-dribble delays, empty or malformed responses, and connection resets.
Contract details to verify
Injection boundary
Document exactly where the fault enters. A service double tests application behavior deterministically, while a controlled proxy can exercise more of the real transport path. Neither result should be represented as proof about a public provider. Record which DNS, connection-pool, proxy, and adapter layers the experiment includes and excludes.
Steady state and hypothesis
Define steady state through user-observable measures: valid-result rate, error rate, latency percentiles, duplicate-response count, and completion within the action deadline. The hypothesis must name the fault, expected mitigation, and acceptable impact. “Fallback works” is not falsifiable; “a reset before response commit causes one eligible fallback attempt and one valid result before the deadline” is.
Fault matrix
At minimum, separate these scenarios:
an HTTP throttling or selected server-error response before content begins;
inability to connect or resolve the dependency;
a delayed first byte and a long-tail latency sample;
a response delivered too slowly in chunks;
an empty response;
a success status followed by malformed data or connection closure;
local cancellation while the primary attempt is pending;
fallback saturation or policy rejection.
Each case needs its own expected classification. Do not collapse every condition into “provider down,” because the safe routing decision can depend on whether any response has already been committed.
Deadlines and the response-commit point
Verify connect, first-byte, inactivity, and overall action deadlines independently where your client exposes them. Streaming and non-streaming paths need separate assertions; the streaming and non-streaming timeout checklist explains why one timeout policy is insufficient.
Choose and instrument a response-commit point. If content has already been shown to a user, silently replacing it with a second response can create duplication or contradiction. Route that condition through an explicit partial-result policy and classify partial success before retrying .
Routing and attempt policy
Confirm which error categories permit fallback, which route is next, and which conditions stop the chain. Assert a maximum total attempt count per action, not merely a per-route retry count. Include a test in which the fallback also fails so the suite can detect loops, route oscillation, or an accidental third attempt.
Output, tool, and safety compatibility
A transport-level success is not enough. Validate the response shape, structured-output requirements, tool-call contract, content policy checks, and application-specific quality floor expected by the original user action. Safety and validation controls must run on the fallback result as well as the primary result; use the fallback safety-check continuity guide when defining these assertions.
Cancellation, capacity, and spend
Cancel a synthetic user action during both the primary and fallback phases. Verify that no new attempt starts after cancellation and that in-flight work is stopped where the integration supports it. Cap experiment concurrency, total actions, fallback attempts, and estimated spend. A technically correct route switch still fails the experiment if it exceeds an agreed operational budget.
Cleanup and evidence ownership
Name the operator authorized to stop the run and the person responsible for confirming recovery. Cleanup must remove the injection rule, temporary cohort, routing override, and experiment-specific alerts. Preserve the scenario version, sanitized logs, metric window, result, and unresolved findings. Do not promote a failed scenario into a deployment gate until the defect is fixed and the same experiment passes.
Failure modes
The selector is broader than intended. A missing cohort constraint exposes unrelated traffic. Default the injector to off, require an explicit synthetic selector, and cap the number of affected actions.
The control and faulted runs use different configurations. This makes the comparison meaningless. Freeze the adapter, routing policy, prompt template, and contract version for both runs.
The test checks only status-code fallback. Real disruptions can appear as latency, DNS failure, connection reset, an empty response, or malformed data after an apparent success. Exercise transport and response-commit boundaries as separate cases.
A partial stream triggers a second visible answer. The router reports recovery, but the user sees duplicated or conflicting output. Assert response-commit state and a single terminal outcome.
The fallback fails and routing loops. A primary-to-fallback test can pass while a two-route failure oscillates. Inject failure on both routes and assert the action-level attempt ceiling.
The fallback violates a semantic contract. The response arrives on time but breaks structured output, tool behavior, or a safety requirement. Keep these checks in the pass criteria.
The experiment outlives its window. A fault rule, route override, or temporary alert remains active. Treat cleanup and the post-run healthy control as mandatory test stages.
Logs become a new data risk. Capturing full prompts, responses, headers, or query strings creates unnecessary exposure. Store categorical outcomes and application-created correlation IDs instead.
An alarm exists but cannot stop the run. Rehearse the abort path with a harmless threshold before injecting a consequential fault.
FAQ
Does fault injection have to run in production?
No. As a conservative staged safety recommendation for this workflow, begin in a non-production environment that resembles the relevant production path, using synthetic traffic and the same adapter and policy configuration where practical. That sequence is this article’s recommendation, not an AWS requirement. AWS says experiments should run in environments that are in or as close to production as possible and emphasizes clear scope, rollback, guardrails, and stop conditions. The Principles of Chaos Engineering prefers production traffic for authenticity while also requiring a minimized blast radius. Move to a production canary only through separate approval, with a much smaller cohort, proven stop controls, and clear ownership.
Is returning one server error enough?
No. It proves only one branch of the classifier. Add connection failure, DNS or access failure, first-byte delay, long-tail latency, slow chunks, empty response, malformed data, and reset scenarios. Test faults before and after the response-commit point because they should not automatically share the same fallback behavior.
What should block a deployment?
Block when an established regression test violates its action deadline, attempt ceiling, single-response rule, output contract, safety checks, cancellation behavior, spend boundary, abort control, or cleanup requirement. Also block if the evidence is too incomplete to determine which route and policy produced the result.
How often should the experiment run?
Run it after changes to routing policy, adapters, timeouts, response parsing, model aliases, fallback eligibility, cancellation, or safety checks. Once a scenario is stable and safe, run it periodically or as an appropriate deployment gate. AWS guidance recommends maintaining successful experiments as automated regression tests.
Should operators store prompts to diagnose a failed run?
Not by default. A synthetic fixture can be identified by a short scenario name and version. Store routing, timing, error, commit-state, usage, and outcome metadata. If content inspection is genuinely required, handle it through a separately governed process rather than general operational logs.
What if the primary response has already started?
Do not assume transparent fallback is safe. Apply the response-commit policy, terminate or label the partial result as your product contract requires, and prevent a second visible answer unless the interface explicitly supports recovery. Test this behavior separately from a failure that occurs before any content is committed.
Reader next step
Choose one low-risk scenario: a synthetic action, one primary route, one eligible fallback, and a bounded delay before the primary response begins. Write the hypothesis, attempt ceiling, action deadline, spend cap, abort condition, expected sanitized fields, and cleanup owner. Run the healthy control, run the faulted path, remove the fault, and run the control again. Only then add connection-reset, malformed-response, and slow-chunk cases.
If you are evaluating CometAPI for the gateway layer, Start with CometAPI , then place the experiment at a boundary you control and verify the exact routes and contracts your application uses.