Redact prompts, credentials, and user data before CometAPI fallback telemetry leaves the gateway.

Last reviewed: 2026-08-10

Direct answer

Build CometAPI fallback telemetry as a new, minimal event from a fail-closed allowlist at the gateway. Do not serialize a request or response object and try to clean it afterward. Drop prompts, completions, tool arguments, tool results, uploaded content, raw request and response bodies, authentication material, cookies, and unapproved user attributes before a log, span, metric label, queue item, crash report, or support export can be created.

Keep only fields needed to answer an operational question: an opaque operation ID, an opaque trace ID, a route template, an approved model label, the fallback attempt number, a bounded reason and outcome, an HTTP status when present, latency, retry classification, redaction-policy version, and counts of fields removed or masked. Mask a value with [REDACTED] only when operators need to know that a field was present. Hash an approved tenant or user correlation value only when stable grouping is necessary, after a privacy review, using a scoped keyed digest and a documented rotation policy. Dropping is the default for credentials and free-form model content.

This boundary belongs in the gateway because every downstream copy increases exposure. A collector-side processor is still useful as defense in depth. The OpenTelemetry redaction processor documentation describes an allowlist that removes attributes not explicitly permitted, blocked key and value patterns, masking or hashing, and diagnostic counts. It should catch mistakes; it should not be the first place raw prompts or credentials are handled.

Use this happy-path operator workflow:

1. The gateway decides that an eligible request needs CometAPI fallback while the raw request remains only in the request-processing path.
2. The network client reads the CometAPI credential from approved server-side storage. The telemetry builder never receives that value. This separation follows the CometAPI quick start , which directs users to server-side environment storage or a local environment file and warns against exposing real keys in repositories, frontend code, screenshots, or support tickets.
3. The gateway creates a fresh event from the allowlist. It maps errors to bounded enums, records a route template instead of a full URL, and records neither headers nor bodies.
4. If an approved free-form operator note is truly required, inspect and replace configured sensitive-data types before adding it. Prefer a local or in-gateway detector for a strict gateway boundary. The Google Cloud sensitive-text redaction guide demonstrates the pattern of inspecting configured information types and replacing detections with placeholders, but sending text to a hosted inspection service is itself a separate data disclosure that requires approval.
5. Validate the event against a closed schema, reject unknown keys, then pass it through a second redaction processor before batching and export.
6. Confirm that dashboards and alerts work from the sanitized event and that incident packets are generated only from the sanitized store.

A sanitized error-path fallback event can look like this:

{
  "event_name": "llm_fallback_attempt",
  "event_schema": 3,
  "occurred_at": "2026-08-10T00:00:00Z",
  "operation_id": "op-4f2a",
  "trace_id": "tr-7c1d",
  "route_template": "/v1/chat/completions",
  "fallback_provider": "cometapi",
  "model_label": "approved-chat",
  "attempt": 1,
  "outcome": "upstream_timeout",
  "http_status": 504,
  "latency_ms": 1800,
  "retry_class": "transient",
  "redaction": {
    "policy_version": "telemetry-v3",
    "fields_dropped": 7,
    "values_masked": 1,
    "marker": "[REDACTED]",
    "inspection_status": "passed"
  }
}

On the error path, fail closed for telemetry export. If inspection times out, the schema sees an unknown field, or the redactor returns an error, discard the candidate telemetry event rather than enqueueing a raw copy. Increment a coarse, predeclared local counter such as telemetry_export_blocked, alert on that counter, and preserve the user-request behavior defined by the service policy. Failing closed for telemetry does not automatically require failing the user’s model request; it means sensitive telemetry cannot bypass the control because observability is degraded.

Who this is for

This policy is for reliability, platform, security, privacy, and observability engineers who operate a gateway that can send LLM traffic to CometAPI during fallback. It is especially relevant when the gateway fans telemetry into multiple destinations: application logs, OpenTelemetry collectors, metrics backends, error trackers, data warehouses, incident chat tools, and vendor support systems.

The owner should be able to change the gateway serializer, the telemetry schema, and collector ordering. On-call operators need the resulting fields to distinguish a primary failure from a fallback failure without seeing customer content. Security and privacy reviewers need an explicit field-by-field contract, not a promise that a broad regular expression will catch everything.

Key takeaways

• Construct telemetry from an allowlist; never clone the live request, response, exception, or SDK object.
• Drop credentials, prompts, completions, tool payloads, headers, cookies, raw bodies, URL queries, URL fragments, and direct user data before serialization.
• Use masking only to communicate presence, and use stable hashing only for a reviewed correlation requirement. Neither is a reason to retain model content.
• Treat collector redaction as a second control. Put it before batching, logging, tail sampling that copies attributes, and every exporter.
• If redaction cannot prove an event safe, block that event’s export and emit only a coarse safe-health signal.
• Build incident and support packets from sanitized storage. Do not let an export script reach raw gateway buffers.
• Test the happy path and deliberately inject forbidden keys, nested objects, encoded text, scanner timeouts, and exporter failures.

Sources checked

• The CometAPI quick start says to keep API keys in server-side environment storage or a local environment file and not expose real keys through repositories, frontend code, screenshots, or support tickets.
• The OWASP Logging Cheat Sheet explains that logging should be designed for an explicit purpose, that externally sourced event data is untrusted, and that sensitive material such as access tokens, passwords, and personal data should be removed, masked, sanitized, hashed, or encrypted rather than logged directly.
• The OpenTelemetry Collector redaction processor documents fail-closed attribute allowlists, blocked key and value patterns, masking or hashing, URL sanitization, and redaction summaries.
• The Google Cloud guide to redacting sensitive data from text documents inspecting text for configured sensitive-data types and replacing detections with a chosen placeholder.

Together, these sources support a layered design: keep credentials server-side, minimize application logging, permit only known telemetry attributes, and inspect any exceptional free-form text. They do not replace a data inventory or a review of the gateway’s actual libraries and export destinations.

Contract details to verify

Start with a field disposition table and make it executable as a schema:

Then verify these boundaries in the deployed system:

• Gateway construction: the telemetry builder accepts individual safe scalars, not a request, response, context, exception, or client object. Unknown fields are rejected.
• URL handling: only a route template is retained. Query strings and fragments are removed before logging, tracing, error reporting, or metric labeling.
• Exceptions: SDK and HTTP-client exceptions are mapped to bounded error classes. Check nested causes, debug formatters, automatic request snapshots, and stack-local capture.
• Streaming: partial output buffers and stream events use the same policy. Cancellation and timeout paths must not dump accumulated content.
• Propagation: trace baggage, span events, resource attributes, metric labels, and message-queue metadata have separate allowlists. A safe log schema does not automatically make them safe.
• Collector order: redaction runs before any processor or exporter that can copy or persist attributes. Confirm what the deployed processor version covers; the referenced documentation describes span, log, and metric datapoint attributes, not a blanket guarantee for every telemetry surface.
• Processor maturity: the refetched OpenTelemetry page labels traces beta and logs and metrics alpha. Verify current stability, behavior, and test coverage for the exact collector release before depending on it.
• Diagnostic mode: production summaries should expose counts, not lists of removed attribute names. The processor documentation notes that debug summaries can themselves reveal information.
• Free-form inspection: define detectors, languages, encodings, size limits, timeouts, and the action for inconclusive results. Under a strict boundary, a hosted detector cannot receive raw gateway text without a separate approved transfer.
• Hash governance: document the purpose, input normalization, scope, rotation, access, retention, and deletion behavior. Do not use an unsalted general-purpose digest as a shortcut for consent or minimization.
• Exports and retention: dashboards, archives, incident tooling, and support bundles read only sanitized data and apply appropriate access controls and expiration.

A collector safeguard might begin with this policy shape, adapted and tested against the deployed version:

processors:
  redaction/fallback:
    allow_all_keys: false
    allowed_keys:
      - event_name
      - event_schema
      - operation_id
      - trace_id
      - route_template
      - fallback_provider
      - model_label
      - attempt
      - outcome
      - http_status
      - latency_ms
      - retry_class
      - redaction.policy_version
      - redaction.fields_dropped
      - redaction.values_masked
      - redaction.inspection_status
    summary: info

An allowlist is the decisive control here: a newly introduced attribute is removed until someone reviews and admits it. A separately tested value scanner can provide another chance to catch unsafe content, but a denylist alone cannot enumerate every way customer content appears.

Failure modes

• Object spread leaks: a convenient serializer copies the complete request and adds an outcome field. Nested messages and headers leave the process before the redactor sees them.
• Exception leakage: an SDK error contains a request snapshot, response excerpt, full URL, or headers. Logging the error object bypasses the normal event builder.
• Wrong pipeline order: batching, tail sampling, debug logging, or an exporter runs before redaction, creating an unredacted copy even if the final backend view looks clean.
• Fail-open inspection: the text detector times out and the application keeps the original note. Safe behavior is to omit the note or block the event.
• Regex-only confidence: patterns miss nested, encoded, segmented, misspelled, or previously unknown personal data. Schema minimization and field dropping must carry the policy.
• Hash overreach: teams hash an email or account identifier and treat the result as harmless. Stable values remain linkable, and predictable inputs may be testable by an attacker with access to the data.
• High-cardinality labels: a prompt fragment, user value, or raw error becomes a metric label or span name. This can disclose data and destabilize the observability backend.
• Diagnostic self-leak: a redaction summary lists the exact forbidden key names. Counts are usually enough for production monitoring.
• Support-path bypass: an incident script attaches raw logs, a crash dump, or an in-memory request snapshot even though dashboards are sanitized.
• Local spill before export: stdout, temporary files, dead-letter queues, or crash handlers persist the candidate event before the export redactor runs.
• Schema drift without rejection: a new SDK or gateway release adds a field and the serializer accepts it by default. Contract tests should prove that unknown fields are removed or rejected.

Test these cases in staging with synthetic markers, never real customer data. Assert absence across gateway stdout, collector debug output, queues, traces, metrics, archives, alert payloads, and generated support bundles. Monitor removal counts for change, but do not turn those counts into a covert list of what was removed.

FAQ

Does fail-closed telemetry mean the user request must fail?

No. The telemetry event and the model request are separate decisions. If redaction or schema validation fails, block or discard the telemetry event and raise a safe health signal. Whether the model request proceeds is determined by the gateway’s availability and risk policy.

Is replacing every detected value with [REDACTED] sufficient?

No. Credentials and raw model content should normally be dropped, so neither the sensitive value nor an unnecessarily revealing field name propagates. Masking is useful when the presence of a field is operationally meaningful and approved.

Should the gateway hash user and tenant identifiers?

Only when stable grouping is necessary and documented. Prefer an operation-scoped opaque ID when possible. If cross-event correlation is required, review the input, scope, rotation, retention, access, and deletion behavior; do not assume hashing makes personal data anonymous.

Can we enable prompt logging temporarily for debugging?

Do not route prompts through the normal fallback telemetry pipeline. If a separate content-capture capability is ever justified, it needs an explicit purpose, authorization, customer and legal review where applicable, strict access, short retention, and its own audited boundary. The fallback log schema should remain content-free.

Is the OpenTelemetry redaction processor enough by itself?

No. It is a valuable second layer for allowed attributes and blocked patterns. The gateway should prevent raw sensitive fields from becoming telemetry at all, and operators should verify processor coverage and ordering for the exact deployed release.

What evidence can on-call staff safely send to support?

Use timestamps, opaque operation and trace IDs, route templates, approved model labels, attempt numbers, bounded outcomes, statuses, latency, and policy counters from the sanitized store. The incident escalation evidence guide provides a compatible evidence workflow without relying on customer content.

Reader next step

Inventory every place a fallback attempt can emit data, then implement one allowlisted event schema and an automated forbidden-field test before the next incident. Compare the result with the reviewable CometAPI retry log fields , run both the happy path and the redactor-error path, and confirm that support exports can be built from sanitized storage alone.

When that boundary is tested, Start with CometAPI and keep the same redaction contract attached to every fallback route.