Last reviewed: October 30, 2026 (Asia/Shanghai).

Direct answer

A synthetic LLM probe should answer one narrow question: can this exact route accept a small, known request and return a response that the fallback adapter can parse right now? Keep that check outside the user-path SLO, give it its own credential, concurrency ceiling, deadline, and budget, and record its result with a probe identity that cannot be mistaken for a customer action.

Use a probe pass as route-contract evidence, not as proof of capacity, model quality, uptime, pricing, or a safe failover. The Google SRE monitoring guidance distinguishes black-box checks of externally visible behavior from white-box internal metrics and recommends looking at traffic, latency, errors, and saturation together. A probe observes one controlled slice. Your user-path telemetry tells you whether real work is failing.

For a CometAPI Chat Completions route, build the smallest request supported by the current CometAPI Chat Completions reference : a dedicated test credential, a model selected from the current route configuration, one fixed harmless message, and the response fields your adapter actually needs. Do not copy a user prompt into the probe or infer a provider health endpoint that the documentation does not describe. If the fallback is streaming, probe the streaming contract separately; a non-streaming pass does not establish streaming readiness.

The safe decision is a conjunction: current user-path evidence shows a problem, the probe has recent contract evidence for the intended destination, the destination still has admitted capacity, and the request is eligible for replay under your retry and data-handling policy. If any part is missing, hold or degrade according to the documented product behavior.

Who this is for

This guide is for platform engineers, SREs, and on-call owners who operate an LLM gateway with a primary route and a fallback route. It fits systems that use a CometAPI-backed fallback, but the separation rules apply to any provider.

It is especially useful when an on-call dashboard has one green health-check tile and one red user-error graph. Those signals can both be correct: a small probe may succeed while real requests are rejected for quota, payload size, model capability, tenant policy, or saturation. The goal is to make the distinction visible before an automated router turns a narrow observation into a broad traffic decision.

This is a design and review workflow. It does not report a production outage, an authenticated provider test, a latency result, or a capacity guarantee.

Key takeaways

  • Give probes a separate identity, credential, queue, deadline, and spend allowance.
  • Keep probe results out of the customer SLO denominator and out of automatic provider-health aggregates used for user traffic.
  • Assert the exact request and response contract that the fallback adapter will use.
  • Join probe and user telemetry for diagnosis, but never let a probe pass erase current user-path failures.
  • Treat probe timeouts, authentication errors, malformed responses, and missing telemetry as explicit outcomes.
  • Use a recent probe only as one input to a bounded, capacity-aware fallback decision.

Sources checked

These primary public references were fetched successfully on October 29 UTC and October 30 in Asia/Shanghai. They are maintained technical references for an evergreen operations task; this article does not claim a new provider release or incident.

Define the probe contract

Write the probe contract before writing a scheduler. It should name the destination route, endpoint family, model reference, request fixture, expected response fields, maximum request duration, maximum response size, and the action taken after each result. A contract can be as small as: “The adapter can send one non-streaming Chat Completions request and parse the required top-level object and text field.” If your fallback is streaming, use the streaming request and assert the terminal event behavior that your adapter requires; a non-streaming pass does not establish streaming readiness.

Use a dedicated credential or scope where the provider and your account policy permit it. The credential should have only the access needed for the probe. Keep its rotation and revocation path separate from the credential serving user traffic so an experiment cannot silently become a production dependency. Store the model identifier in trusted route configuration and refresh it when that configuration changes. A probe must not select an arbitrary model because a catalog lookup happens to return one.

Use a fixed, short, non-sensitive fixture. The fixture should exercise the parser without asking for a realistic customer task. For example, a test message can ask for a single predetermined word. Keep a fixture version beside the route and adapter version. Changing the endpoint family, output mode, tool schema, or parser while retaining the old probe result creates false confidence.

The endpoint-health workflow already in this site’s corpus covers the separate question of checking Chat Completions and Responses contracts. Use this article’s boundaries to keep that evidence from being confused with user-path availability.

Isolate resources and side effects

Put probes on a bounded queue with a small concurrency limit and a finite deadline. The Google SRE overload guidance describes health checking as work that can consume resources, especially when many low-volume clients each perform their own checks. A frequent probe can therefore make an overloaded route look busier while providing little additional information. Choose a minimum interval, a maximum in-flight count, and a stop switch.

Keep probe traffic out of the customer queue when possible. If the transport or provider pool is shared, reserve a small explicit slice or admission token so probe retries cannot consume capacity needed by user actions. Never let a probe failure start another probe recursively. A scheduler may retry a transient probe failure under its own small budget, but the event must retain the attempt number and stop at the configured ceiling.

Do not allow a probe to execute tools, write user state, send notifications, update a production cache, or create durable workflow jobs. A model response that proposes an action is still probe output. Route it to a record-only sink or use a fixture and adapter that cannot reach side-effecting tools. If that isolation cannot be proven for a request class, exclude the class from live probes and use an offline evaluation set.

Keep the probe response out of the user response path. The probe scheduler can publish a low-cardinality result such as contract_pass, auth_failure, timeout, parse_failure, or capacity_rejected; it should not publish prompt or completion content to a general log stream.

Keep two signal families

Maintain separate measurements for probe outcomes and user outcomes. A probe event describes the route and fixture at a point in time. A user event describes a real action, its final contract result, and its latency or cancellation state. Do not add probes to the denominator of a customer availability SLO, and do not use successful probes to offset user failures. The error-budget burn guide explains why an SLO should count the completed user action rather than each physical provider attempt.

Join the families in an investigation view using route, endpoint family, model class, region bucket, configuration version, and time. Avoid putting request IDs, tenant IDs, prompts, or full model names with unbounded values into metric labels. The OpenTelemetry HTTP semantic conventions provide stable HTTP attributes for request method, status, duration, and error information; use your tracing system for a high-cardinality correlation identifier rather than a metric dimension.

A sanitized event can look like this:

event: llm_route_probe
probe_id: probe-20261030-001
route: fallback-cometapi
endpoint_family: chat_completions
fixture_version: text-v2
route_config_version: fallback-v7
credential_scope: probe-only
http_status_class: 2xx
contract_result: pass
latency_ms: 640
request_bytes_bucket: small
response_bytes_bucket: small
error_type: null
user_slo_event: false
fallback_promotion: held
content_logged: false

The event records evidence about the probe, not a claim that all user requests will succeed. Keep the exact probe response and credential out of ordinary logs. If an approved debugging process needs content, retain the minimum protected fixture and response reference for the shortest permitted period.

Make fallback admission explicit

A router should consume a probe record through a bounded freshness rule. For example, it may require a successful contract probe within a short configured interval, an available destination admission token, and no current authentication or policy failure. Those are application rules, not provider guarantees; choose values from your own deadline, traffic, and quota contracts.

Then evaluate the user action. Check cancellation, remaining deadline, request size, model capability, data-residency policy, and whether the first attempt may already have executed. The request-replayability guide explains why a fresh probe pass cannot make an ambiguous or non-replayable user request safe to repeat.

Use a decision record rather than a boolean health flag:

user_action: action-42
primary_outcome: timeout_after_transmission
probe_evidence: contract_pass_38s_old
destination_capacity: admission_unknown
request_replayable: true
remaining_deadline_ms: 2100
fallback_decision: hold_for_reconciliation
reason: probe_pass_does_not_prove_capacity_or_non_execution

A probe pass with unknown capacity should not open the floodgates. A probe failure caused by its own expired credential should not automatically mark the provider globally unhealthy. Preserve the failure class and scope the effect to the route, credential, endpoint family, and configuration version that were actually tested.

Verify the failure paths

Exercise at least these controlled cases in a non-production environment: a successful contract response, an invalid probe credential, an invalid model or request fixture, a response that the adapter cannot parse, a deadline after headers, a queue at its probe concurrency ceiling, and a disabled probe scheduler. Confirm that each case produces one bounded event and that no case reaches a user response or side-effect executor.

Also test the signal split. Inject a probe success while user traffic returns authentication failures, quota rejections, or malformed responses. The user-path signal must remain bad, and fallback admission must require the separate destination-capacity and replay checks. Inject a probe failure while user traffic is healthy and confirm that the system records a route-contract warning without taking healthy primary traffic offline.

These are verification steps, not reported test outcomes. Record the client library, route configuration, fixture version, status class, normalized error class, and decision owner. The controlled fault-injection guide can supply the broader failover exercise after the probe boundary is in place.

Contract details to verify

Treat the probe as a small operational contract with named inputs, limits, evidence, and decisions. A passing HTTP request is only useful when the route and adapter that will receive fallback traffic are the same ones the probe exercised.

Route and request identity

Record probe_id, route name, endpoint family, route-configuration version, model reference, fixture version, credential scope, and observed_at. Keep the probe identity separate from a user-action or provider-attempt identifier. For the CometAPI Chat Completions path, verify the configured base URL, POST /v1/chat/completions, authorization placement, required model and messages fields, streaming mode, and the response fields the fallback parser requires against the current API reference . A probe that uses a different endpoint family, model, or stream mode is evidence for that other contract.

Resource and side-effect limits

Set a maximum request duration, response-size limit, queue length, in-flight probe count, retry count, and probe spend allowance. Give the scheduler a stop switch and make exhaustion a visible result. The Google SRE overload guidance explains why health checks themselves consume capacity; use that principle to keep probe admission separate from customer admission when the transport or provider pool is shared. State explicitly that the probe cannot execute tools, write application state, update a production cache, or send a notification.

Result and telemetry schema

Define a finite result set such as contract_pass, auth_failure, invalid_request, timeout, parse_failure, capacity_rejected, and scheduler_disabled. For every result, retain the HTTP status class, duration, normalized error type, route, endpoint family, model class, configuration version, and whether the event counts toward the user SLO. The OpenTelemetry HTTP semantic conventions provide stable HTTP request, response, duration, and error attributes for this record. Keep request IDs, prompts, completions, and credentials out of metric labels and ordinary logs; use a trace or protected reference when high-cardinality correlation is needed.

Freshness and admission rules

Specify when a probe record expires and which configuration changes invalidate it. A router might require a recent contract_pass, a destination admission token, no scoped authentication or policy failure, and a user action that still has time remaining. These are application rules, not provider guarantees. A fresh pass must not override current user errors, prove that capacity is available, or authorize a request whose first attempt may already have executed. Record the complete decision and the reason for holding, degrading, or allowing one bounded fallback attempt.

A compact contract record can look like this:

probe_id: probe-20261030-001
route: fallback-cometapi
endpoint_family: chat_completions
route_config_version: fallback-v7
fixture_version: text-v2
credential_scope: probe-only
max_in_flight: 1
request_deadline_ms: 3000
max_retries: 0
result: contract_pass
observed_at: 2026-10-30T05:39:58+08:00
expires_after_seconds: 60
user_slo_event: false
side_effects: blocked

Before enabling a router dependency on this record, verify the happy path and each bounded failure path with the exact deployed adapter. Confirm that a stale or missing record holds admission, a probe-only credential failure does not trip every route, and a user cancellation or ambiguous first attempt cannot be converted into an automatic replay solely because the probe is green.

Failure modes

  • The probe shares the user budget. A retry loop or a large fixture consumes the same pool as real work. Reserve probe capacity and make the scheduler stop at a hard ceiling.
  • A green tile hides a red user path. The fixture is short or privileged while customers hit a different model, quota, payload class, or region. Segment the evidence and keep user outcomes authoritative for user SLOs.
  • The probe becomes a side-effect path. A tool proposal, cache write, or notification runs under the normal executor. Use a record-only sink or exclude that workload class.
  • A credential failure trips every route. Scope the event to the tested credential and route configuration. Rotation or entitlement problems need their own operator response.
  • A stale pass authorizes a new outage. Attach observed_at, fixture version, and route version to every record. Expire evidence when any contract input changes.
  • Metrics mix probes and users. A passing probe offsets a failed customer action or inflates request volume. Keep the signal families separate and join them only in diagnosis.
  • A probe starts automatic failover. A route-contract check does not establish request replay safety, destination capacity, or data-policy eligibility. Require the complete decision record.

FAQ

Does a passing probe prove the fallback provider is healthy?

No. It proves only that one bounded request reached the tested route and met the tested contract at the observation time. It does not establish capacity, uptime, model quality, quota, pricing, or success for every user request.

Should probes use the same credential as users?

Prefer a dedicated scope when the provider and account policy support it. If a shared credential is unavoidable, label the scope explicitly and ensure probe volume is included in the same admission and spend budget. Never hide a shared-budget probe behind a separate “free” health metric.

Can a probe use a real user prompt?

No. Use a fixed, harmless, non-sensitive fixture. Real prompts add privacy exposure and make the result hard to reproduce. If realistic request-shape evaluation is needed, use a separately governed shadow or offline evaluation workflow; the shadow-testing guide explains the added side-effect and promotion boundaries.

How often should a probe run?

There is no universal interval. Pick an interval that fits the route’s user deadline, provider quota, and desired evidence freshness. Set a maximum rate and a kill switch, then review probe cost and information value during a controlled rollout.

Should a probe failure page the on-call engineer?

Only after you define the scope and corroborating evidence. A single probe can fail because of its credential, fixture, network path, or scheduler. Page on a customer-impacting signal or a clearly scoped route-contract condition with a runbook, and retain the probe event for diagnosis.

Reader next step

Write a one-page probe contract for one fallback route. Name the endpoint family, model reference, fixture version, credential scope, queue and deadline limits, side-effect guard, event fields, freshness rule, and stop conditions. Run the controlled success and failure cases in isolation, then inject contradictory evidence: a passing probe with failing user traffic and a failing probe with healthy user traffic.

Before allowing the router to use the record, review the resulting events with the owner of the user SLO and the owner of downstream tool execution. Confirm that a probe cannot return content to a user, consume unbounded shared capacity, or authorize an ambiguous replay. Then pair the record with the site’s response-shape review and make the final fallback decision through your normal change process.