Reconcile Token Usage When CometAPI Streams End Early

Last reviewed: September 14, 2026

Direct answer

The safe pattern for CometAPI interrupted stream token usage is to treat delivery completion and accounting completion as separate state machines. Record a provider-confirmed token total only when the provider explicitly emits it. If the stream closes first, leave the confirmed total unset, preserve any provider-reported partial count, store any local estimate in a separate field, and send the attempt to a bounded reconciliation queue.

The CometAPI Chat Completions documentation says that an OpenAI-compatible streaming request can set stream_options.include_usage to true. The resulting usage data appears in a final chunk before [DONE], and that chunk can have an empty choices array. The OpenAI Chat Completions reference adds the crucial failure condition: an interrupted stream may not deliver that final usage chunk.

Do not reduce all of those observations to one complete flag. Keep at least these two dimensions:

• stream_status: open, complete, client_canceled, upstream_eof, timed_out, provider_error, or parser_error.
• usage_status: not_seen, provider_partial, provider_confirmed, estimated, or unresolved.

These dimensions can legitimately disagree. A final usage event can be persisted and then the connection can disappear before the terminal marker, leaving confirmed usage on an abnormal stream. Conversely, a terminal marker can be observed without a stored usage event because of a parser defect or contract anomaly. A stream can also deliver useful text to the client while its accounting remains unresolved.

The central rule is simple: never copy an estimate into a provider-confirmed field. Keeping provenance explicit lets operators repair accounting without rewriting history or pretending that emitted text is the whole billable workload.

Who this is for

This guide is for platform engineers, gateway owners, FinOps teams, and on-call responders running metered LLM streams through CometAPI. It is especially relevant when the gateway forwards output immediately, supports more than one provider protocol, or uses token limits for tenant quotas, spend protection, or fallback admission.

It assumes the application already has a durable request or attempt record. If it does not, add that record before relying on streaming telemetry for billing, quotas, or incident analysis.

Key takeaways

• Request terminal usage metadata when the route supports it, but design for that metadata to be absent.
• Persist stream state and usage state independently.
• Treat a provider-emitted final total as confirmed, a cumulative midstream value as partial, and a locally calculated count as an estimate.
• Parse usage-only chunks before indexing choices[0].
• Never turn missing usage into zero; zero is a measurement, while missing is an evidence state.
• Reconcile by attempt, then aggregate attempts under the logical user action so retries are not silently double-counted.
• Do not replay a completed or partially completed generation merely to recover its accounting metadata.

Sources checked

The implementation guidance below is grounded in four public protocol references:

• CometAPI: Create a chat completion documents SSE delivery, the optional final usage chunk, its empty choices array, and the [DONE] terminator.
• OpenAI: Create chat completion documents stream_options.include_usage and warns that interruption can prevent receipt of the total-usage chunk.
• Anthropic: Streaming messages documents named SSE events, cumulative usage in message_delta, in-stream errors, and the need to tolerate new event types.
• Google: Generating content defines usage metadata categories including prompt, candidate, cached-content, thought, and total token counts.

Contract details to verify

Separate evidence from interpretation

A cross-provider ledger needs to retain what was observed before it decides what that observation means.

The Anthropic streaming contract says usage in message_delta is cumulative. Summing successive values would therefore inflate usage. It also permits error events inside an SSE connection, so an initial successful HTTP response does not prove that the generation completed.

Google’s documented metadata provides another warning against one generic output counter. Prompt, generated candidates, cached content, and model thoughts can have distinct fields. A local counter derived from visible text cannot be assumed to include every provider-reported category. Preserve the native categories, then map them into a canonical view without discarding the original field names or provenance.

Use an explicit ledger schema

A useful attempt record has five groups of fields:

1. Identity: a short internal request reference, attempt number, route, and selected model.
2. Delivery: stream status, last complete event type, event sequence, bytes forwarded, and whether the downstream client disconnected.
3. Usage: prompt, output, cached, reasoning or thought, and total counts when the protocol supplies them.
4. Provenance: usage status, usage source, protocol, and whether a value is cumulative, final, or estimated.
5. Failure evidence: a normalized transport class or provider error type, without prompt content or sensitive headers.

A sanitized log record can look like this:

{
  "request_id": "req-42",
  "attempt": 1,
  "route": "chat-completions",
  "model": "selected-model",
  "stream_status": "complete",
  "usage_status": "provider_confirmed",
  "usage_source": "terminal_usage_event",
  "prompt_tokens": 418,
  "output_tokens": 96,
  "cached_tokens": 120,
  "reasoning_tokens": 0,
  "total_tokens": 514,
  "last_event_type": "done",
  "last_event_sequence": 19,
  "bytes_sent": 2840,
  "client_disconnected": false,
  "transport_error_class": null,
  "provider_error_type": null,
  "estimate_method": null
}

Treat cached, reasoning, and other detailed counts according to the source protocol rather than automatically adding every field into total_tokens. Some fields are components or subsets. Validate nonnegative values and documented relationships, but retain the provider’s total as reported.

Keep prompts, generated text, tool arguments, cookies, authentication material, and complete request headers out of the accounting log. Allowlist route and model labels instead of copying arbitrary inbound values. If tenant correlation is necessary, use an internal pseudonymous reference with a documented retention period.

Happy-path operator workflow

1. Before opening the upstream connection, write a durable attempt row with stream_status=open and usage_status=not_seen. Associate it with the logical user action, but give each retry or fallback its own attempt number.
2. For the documented OpenAI-compatible CometAPI route, request streaming usage with stream_options.include_usage=true.
3. Parse complete SSE frames rather than treating network reads as event boundaries. Accept unknown event types without crashing, while recording that an unfamiliar event occurred.
4. Forward content deltas and update the last-event sequence. Test for a usage object before accessing choices[0], because the documented terminal usage chunk has no choices.
5. When usage arrives, persist the native object or its approved count fields and the normalized fields in one transaction. Set usage_status=provider_confirmed for that attempt.
6. When [DONE] arrives, set stream_status=complete. Do not overwrite a previously confirmed usage state if the terminal-marker update is retried.
7. Emit one idempotent accounting event keyed by the request reference and attempt number. A uniqueness constraint should make duplicate terminal frames harmless.
8. On the normal-path dashboard, confirm that completed streams, confirmed-usage attempts, and accounting events remain aligned. Sample records should show the usage event immediately before the terminal event for this route.

Parser reliability is a prerequisite for this workflow. The site’s stream parser integrity guide explains why arbitrary network chunks must not be mistaken for complete SSE events.

Error-path operator workflow

1. Classify the ending as downstream cancellation, upstream EOF, timeout, in-stream provider error, parser error, or worker shutdown. Do not use a generic failed value when the evidence can distinguish them.
2. Atomically freeze the last complete event sequence, whether any text reached the client, the latest provider-reported usage, and the usage provenance.
3. If the protocol supplied cumulative usage, preserve the greatest valid value observed in event order as provider_partial. Never sum cumulative snapshots.
4. If local policy requires an estimate, write it to separate estimated fields with the counting method and model mapping. Keep the provider-confirmed total null.
5. Enqueue one reconciliation task per attempt. If the deployment has a later authoritative account ledger, compare against it and record the reconciliation source and time. If no such evidence exists, close the task under an explicit unresolved policy rather than fabricating a total.
6. Make reconciliation updates conditional on record version and current state so a late worker cannot replace a newer confirmed event.
7. Alert on the rate and age of unresolved attempts, not merely their count. Operators should inspect whether failures cluster by route, model, client network, deployment, or parser version.
8. Keep retry policy separate from accounting repair. Missing usage alone is not a reason to generate the answer again. If a retry is appropriate for delivery, apply the site’s guidance on bounding fallback attempts and account for every attempt separately.

Test the contract before rollout

Run deterministic tests that cover a normal terminal usage chunk, an empty choices array, EOF after several text deltas, EOF after usage but before [DONE], [DONE] without usage, an in-stream error, a downstream cancellation, duplicate terminal events, and an unknown SSE event type. Each test should assert both state dimensions and the number of accounting events written.

Also test process failure between receiving usage and committing it. The event handler and storage design should either commit once or safely replay the same event. A parser test that only checks rendered text will miss the accounting failure this article is designed to prevent.

Failure modes

• The parser indexes an empty choices array. The final CometAPI usage chunk is valid even though choices is empty. Code that reads choices[0] first can throw away the most important accounting event.
• The gateway equates [DONE] with confirmed usage. The marker and usage chunk are separate evidence. Store separate flags and alert when the expected pair is incomplete.
• A successful HTTP status masks an in-stream error. Anthropic documents errors inside the SSE stream. The gateway must inspect event types until the stream ends.
• Cumulative values are added as deltas. Summing Anthropic message_delta usage snapshots can multiply the recorded count. Replace the previous snapshot in event order.
• Missing becomes zero. Zero implies the provider measured no tokens. An absent terminal event means the system does not know the final count.
• Visible output is treated as the complete workload. Provider metadata can distinguish prompt, cached, generated, and thought-related counts. Text observed by the client is only one piece of evidence.
• Retries share one accounting row. A fallback can overwrite the primary attempt or cause both attempts to be billed as one. Preserve an attempt ledger beneath the logical request.
• Client cancellation and upstream failure are conflated. The corrective actions differ. Record which side closed and whether the gateway continued consuming the upstream stream.
• A worker dies before durable persistence. Text may already have reached the client while no terminal usage record survives. Commit event progress durably and make terminal handling idempotent.
• Late reconciliation wins a race. A delayed estimate or external comparison can overwrite provider-confirmed data unless updates check provenance, version, and current state.
• The unresolved queue grows silently. Track unresolved rate, oldest age, and value at risk. Apply a documented closure policy and escalate sustained increases.

FAQ

Does [DONE] prove that token usage was recorded?

No. In the documented CometAPI sequence, the final usage chunk precedes [DONE], but they remain separate events. Persist each observation independently. Seeing [DONE] without a usage record should produce a contract-mismatch or parser-health signal when usage was requested.

Can the gateway count streamed text locally instead?

It can produce a clearly labeled estimate, but that estimate should not masquerade as a provider total. Documented provider metadata can contain input, cached-content, generated-output, and thought-related categories that are not recoverable from visible text alone.

Should missing usage trigger an automatic retry?

Not by itself. A new attempt can create additional output and additional usage while leaving the original attempt unresolved. Retry only when the delivery policy calls for it, and maintain separate accounting for both attempts.

How should cumulative usage events be normalized?

Retain the latest valid cumulative snapshot in event order. Do not sum snapshots. If the expected terminal lifecycle event never arrives, keep that value as provider-reported partial evidence unless the protocol contract establishes it as final.

What if usage arrives after the downstream client disconnects?

If policy allows the gateway to keep consuming upstream events, it can still persist the provider’s final usage while marking delivery as client-canceled. If the gateway cancels upstream immediately, the attempt may remain unresolved. Whichever behavior you choose should be explicit, tested, and visible in the ledger.

What should operators measure?

Track the percentage of attempts with provider-confirmed usage, unresolved attempts by failure class, reconciliation latency, duplicate accounting events rejected, and mismatches between complete-stream and confirmed-usage rates. Break those metrics down by route, model, and deployment version without storing request content.

Reader next step

Start with one streaming route. Enable its documented usage option, add independent stream and usage states, and run the interruption tests before using the ledger for quotas or chargeback. Then set an alert on unresolved usage age and review every automatic retry path to ensure attempts cannot overwrite one another.

Pair the ledger with token-aware fallback admission so uncertain prior usage is considered before another model call begins.

If you want an OpenAI-compatible route on which to apply this workflow, Start with CometAPI .