Asynchronous LLM work moves the reliability boundary away from the request socket and into an event pipeline. A caller can disconnect, a provider can retry a notification, and a completion can become available after the original request has timed out. The safe unit of work is therefore not an HTTP delivery. It is a verified event recorded once, reconciled against the provider resource, and connected to side effects through an idempotent workflow.

Last reviewed: 2026-08-25

Direct answer

Build an LLM completion webhook inbox with five layers: authenticity, durable admission, asynchronous processing, authoritative reconciliation, and idempotent effects. Accept the request body as raw bytes, verify the provider signature and timestamp according to that provider’s contract, and derive a stable identity from the documented event identifier. OpenAI’s current Webhooks guide specifically recommends a fast successful response, background processing, and the webhook-id header as an idempotency key when duplicate events arrive.

After verification, insert an inbox row in the same transaction that records the event’s identity and a digest of the body. Put a uniqueness constraint on (provider, stable_event_id). If the insert succeeds, acknowledge with a 2xx response and enqueue work. If the uniqueness check reports an existing row, acknowledge the duplicate without running the business effect again. Do not make a model retrieval, notification, billing action, or fallback decision on the request thread.

The worker should retrieve or poll the provider’s completion resource and compare it with your own request ledger. A webhook says that something changed; it does not, by itself, prove that the output you intend to expose is still available, complete, or compatible with the original request. Apply a monotonic state transition such as accepted to running to succeeded, and reject a late event that would move a terminal record backward. Emit external effects through an outbox keyed by the same logical operation, so a worker retry cannot send two notifications or start two fallback generations.

This design is intentionally provider-neutral. The Standard Webhooks specification recommends signing the payload together with delivery metadata, checking timestamp tolerance, and using the webhook identifier for deduplication. The provider adapter owns signature canonicalization and event parsing; the inbox, state machine, reconciler, and outbox remain shared.

raw request bytes
  -> verify signature and timestamp
  -> insert provider plus stable event identity
  -> acknowledge 2xx after durable insert
worker
  -> retrieve or poll the completion resource
  -> reconcile against the request ledger
  -> advance state monotonically
  -> enqueue an idempotent outbox effect

Who this is for

This pattern is for teams operating an API gateway, job service, agent runtime, or batch coordinator that accepts work from users and receives a later model completion. It is especially useful when a response can take longer than a client connection, when more than one provider is available, or when a completion can trigger an email, tool call, database write, moderation decision, or fallback request.

It is not necessary for a strictly synchronous prototype that discards work when the caller disconnects. It becomes necessary as soon as the product promises eventual completion, retries a provider call, or lets an operator replay an event. SREs can use the inbox as the unit for backlog and replay metrics; platform engineers can keep provider quirks behind adapters; application engineers can consume a stable state transition instead of interpreting five different webhook payloads.

Key takeaways

  • Verify the raw body before parsing or queuing it. A parsed-and-reserialized body may no longer match the signed bytes.
  • Persist the event identity before returning success. An in-memory dedupe set is lost during a restart and cannot protect a second worker.
  • Use the provider’s documented stable identifier, not only a resource ID. Several legitimate events can refer to one completion resource.
  • Treat delivery as at-least-once and unordered. Stripe’s webhook documentation explicitly describes duplicate deliveries and says event order is not guaranteed; that is a useful design assumption even when another provider’s current behavior appears ordered.
  • Reconcile the resource before exposing output or invoking a fallback. A completion may be partial, canceled, expired, or no longer retrievable.
  • Keep logs useful without copying prompts, generated text, raw headers, customer identifiers, or credentials.
  • Make the outbox effect idempotent separately from inbox ingestion. Exactly-once delivery is not a prerequisite for exactly-once business intent.

Sources checked

The design is based on the following refetched public documentation, with provider-specific behavior kept separate from general engineering guidance:

  • OpenAI Webhooks documents response.completed notifications, signature verification, quick 2xx acknowledgement, retries for up to 72 hours with exponential backoff, possible duplicate copies, and webhook-id deduplication.
  • OpenAI Background mode documents asynchronous execution, polling through queued and in_progress states, terminal states, and temporary retention of responses when store=false.
  • Standard Webhooks specification describes signed metadata, stable event identifiers across retries, timestamp tolerance, and jittered retry guidance.
  • Stripe webhook documentation provides an independent production example of raw-body verification, fast 2xx acknowledgement, duplicate delivery handling, unordered events, and multi-day retries.
  • Anthropic Batch processing documents independently processed requests, polling to an ended state, partial results after cancellation, 24-hour batch expiry, and a longer results availability window.

These sources do not define one universal LLM webhook contract. They show why the adapter must be explicit about identifiers, signatures, retry windows, ordering, and retention rather than assuming that one provider’s behavior applies everywhere.

Contract details to verify

Event identity

Write down which identifier is stable across retries. For OpenAI, the guide names webhook-id as the idempotency key. The Standard Webhooks guidance also describes an event ID that remains the same when a failed delivery is retried. Your adapter should expose one normalized stable_event_id and retain the original provider fields for diagnostics. If a provider gives both a payload event ID and a header ID, record both and document which one drives the uniqueness constraint.

Use a composite key that includes the provider or endpoint configuration. The same-looking event ID from two providers must not collide. Store a body digest as well: a repeated ID with a different digest is a contract violation or a replay worth investigating, not an ordinary duplicate.

Authenticity and acknowledgement

Capture the raw bytes, signature headers, and receipt timestamp before a framework parses the body. Verify the signature over exactly the bytes the provider sent, then enforce a timestamp tolerance to limit old replays. Never put verification secrets in application logs. Stripe likewise warns that changing the raw request body breaks signature verification, while OpenAI’s examples use an SDK verifier around the unmodified request data.

A successful acknowledgement means only that the inbox has durably accepted the event. OpenAI says a slow or non-2xx endpoint is retried for up to 72 hours. Stripe documents up to three days of exponential retries. Set a short handler deadline and monitor the difference between provider delivery time and worker start time. If authenticity fails, quarantine the request and follow the provider’s documented response policy; do not enqueue a side effect. Returning non-2xx can itself invite retries, so make that choice deliberate and observable.

Resource lifecycle and reconciliation

Keep a request ledger with the local operation ID, provider, model route, request hash, intended output contract, and current state. The worker uses the webhook’s resource ID to retrieve the authoritative object, or polls it if the event is only a notification. OpenAI’s Background mode guide says callers should poll while a response is queued or in_progress and stop at a terminal state. It also says a background response with store=false may be deleted after roughly 10 minutes, so a reconciliation job cannot wait indefinitely.

Do not assume that a batch has one all-or-nothing result. Anthropic says each request in a Message Batch is handled independently, and a canceled batch can contain partial results; its guide also states that batches expire after 24 hours. Track child request state, distinguish succeeded, errored, canceled, and expired, and record whether a result was observed before the resource disappeared. If the provider resource is gone, use the local ledger and inbox evidence to decide whether to mark the operation unavailable, retry a safe read, or ask for operator intervention. Do not blindly regenerate just because retrieval failed.

State and side-effect contract

Represent transitions as guarded writes. A worker may receive completed before started, or a stale failed event after a successful retrieval. Compare version, provider timestamp, or an authoritative status before changing state; when none is available, keep the first terminal decision and quarantine the conflict for review. The exact ordering rule is an application decision, but it must be deterministic.

Separate state transition from effect delivery. A transaction can mark the inbox row processed and create an outbox row with a logical effect key such as operation_id:publish-result. A dispatcher claims the outbox row, performs the effect with an idempotency mechanism where the destination supports one, and records the outcome. A timeout after the destination accepted the effect is then a retry of the same intent, not a new intent.

Sanitized logging contract

Log enough to answer who received what, when, and what decision followed, without recording model content. A useful structured record looks like this:

provider=provider-a
event_type=completion.finished
event_id_hash=sha256:example
resource_id=resp_example
received_at=2026-08-25T00:00:01Z
event_timestamp=2026-08-25T00:00:00Z
signature_result=valid
dedupe_result=new
http_status=204
queue_delay_ms=42
reconcile_result=terminal_success
outbox_result=created
tenant_ref=tenant_hash:example

Keep prompts, output text, tool arguments, raw bodies, full headers, email addresses, and credential material out of ordinary logs. If an incident requires payload inspection, store a separately access-controlled, time-limited artifact and reference it by an opaque case ID. Hash or tokenize tenant and event identifiers where practical, but retain enough stable identity to correlate retries.

Happy-path operator workflow

  1. A provider posts a completion event. The edge handler captures bytes and metadata, verifies the signature and timestamp, and validates the minimum envelope (type, timestamp, and resource identifier).
  2. A database transaction inserts the inbox row and request-ledger observation under a unique provider/event key. The transaction also records accepted_at and a body digest.
  3. The handler returns 204. A queue worker claims the row with a lease, retrieves the resource, and confirms that the resource belongs to the local operation.
  4. The worker applies the guarded state transition, creates one outbox row, and marks the inbox row processed. A dispatcher delivers the user-visible update or other effect and records a receipt.
  5. Metrics show one accepted event, one worker completion, and one effect. A later duplicate finds the existing key, increments duplicate_count, returns success, and performs no second effect.

Error-path operator workflow

For an invalid signature or stale timestamp, write a minimal quarantine record, increment an authenticity alert, and keep the payload out of the worker queue. For a duplicate, do not overwrite the original body digest; compare it and alert if the bytes differ. For an event that arrives out of order, retain it as evidence, retrieve the authoritative resource, and apply only the transition allowed by the state machine. For a missing or expired resource, mark reconciliation as unavailable, preserve the inbox row, and route the operation to a bounded manual or fallback decision rather than starting another generation automatically. For a worker crash after retrieval, let the lease expire and retry the same inbox row; the guarded transition and outbox key prevent duplicate business effects.

Failure modes

The handler parses before verifying

JSON middleware can normalize whitespace, key order, or encoding. Verification then fails even though the provider signed the original bytes, or an implementation may accidentally verify a different representation. Capture and verify first; parse only after authenticity succeeds.

The handler does work before acknowledging

A model retrieval or database call that exceeds the provider’s short delivery deadline causes a retry. The first attempt may still finish, so the second attempt can create duplicate work. Durable admission plus a fast 2xx removes this timing race.

Deduplication uses only the resource ID

One resource can legitimately produce multiple lifecycle events. Keying only on resp_example can discard a meaningful transition. Use the documented event or webhook identifier and retain the resource ID as a secondary index.

A duplicate overwrites evidence

If a second delivery with the same ID replaces the first payload, investigators lose the original signature and digest. Keep the first accepted record immutable, store a duplicate counter, and quarantine a digest mismatch.

Arrival order is treated as truth

Stripe explicitly says event order is not guaranteed. A late failure can therefore arrive after a success. Guard terminal transitions and reconcile the resource instead of applying events as an append-only command stream.

The acknowledgement is not durable

Returning 2xx before the inbox transaction commits tells the provider to stop retrying while the event can still be lost. Acknowledge only after the unique insert is committed and the row is visible to a worker.

Retention windows are mixed up

OpenAI background data with store=false may disappear after roughly 10 minutes, while Anthropic batches can expire after 24 hours and retain results for a different period. A generic retry timer can therefore poll an object that no longer exists or retain sensitive data longer than intended. Store provider-specific deadlines in the operation record and alert before each one.

Side effects are not idempotent

A completion can trigger a message, a tool invocation, or a billing action after a worker timeout. If the destination accepts the first request but the worker never records the response, a retry is unavoidable. Use an effect key and destination-level idempotency where available; otherwise keep a durable send ledger and reconcile manually on ambiguity.

Partial work is collapsed into one result

Independent batch requests can finish in different states. Marking the entire batch successful because one child succeeded hides errors; marking it failed because one child errored can cause successful records to be rerun. Keep child-level status and make replay selective.

Logs become a second data leak

Copying the full webhook body into every retry log increases exposure and makes deletion difficult. Sanitize by default, use hashes and bounded reason codes, and grant payload access only through an explicit incident process.

FAQ

Is receiving a completion webhook proof that the answer is ready?

No. It is a delivery signal. Verify the event, then retrieve or poll the authoritative resource and check its terminal status and your local request contract before publishing output.

Should I use the model response ID as the dedupe key?

Usually not by itself. Use the provider’s documented stable event or webhook identifier, namespaced by provider. Keep the response ID for reconciliation and conflict investigation. If the provider exposes only a resource ID, document the limitation and add a local operation sequence so distinct lifecycle observations are not silently collapsed.

What should happen when the same ID has a different body?

Treat it as a security or provider-contract anomaly. Keep the original accepted record, store the second digest and receipt metadata, quarantine the new body, and alert. Never let the later payload rewrite the state that was already accepted.

How long should inbox rows be retained?

At least through the provider’s retry and replay risk window, plus the period your incident process needs. The exact value is provider- and policy-specific: OpenAI documents a 72-hour delivery retry window, while Stripe documents up to three days. Coordinate retention with payload sensitivity and any provider resource expiry; do not infer one universal number.

Do I need both webhooks and polling?

For important asynchronous work, yes. Use webhooks for low-latency notification and polling or a reconciliation sweep to find missed deliveries, handle out-of-order events, and recover when a provider resource changes state without a notification. Bound the sweep by the provider’s retention deadline.

Does this pattern change when a fallback provider is involved?

The inbox contract should stay the same. Create a separate provider adapter and namespace its event IDs, then record the selected route and fallback reason in the request ledger. A fallback must not bypass signature checks, dedupe, state guards, or the outbox. If the fallback uses a different response schema, reconcile it into the same internal terminal states before exposing it.

Reader next step

Implement the smallest vertical slice before adding more providers:

  1. Create an inbox table with a unique (provider, stable_event_id) key, immutable raw-body digest, receipt timestamps, and a processing lease.
  2. Add a provider adapter that verifies raw bytes and timestamp metadata, returns a normalized event envelope, and never logs credentials or model content.
  3. Return 2xx only after the insert commits; send all non-trivial work to a worker.
  4. Add a request ledger, guarded terminal transitions, a reconciliation read, and an outbox keyed by logical operation.
  5. Test valid delivery, duplicate delivery, altered duplicate, stale timestamp, out-of-order events, worker crash, missing resource, partial batch completion, and a resource-expiry deadline.

For adjacent reliability controls, compare this inbox with the site’s guidance on replayable request bodies and duplicate tool-side-effect safeguards . Then run one controlled replay drill and keep the resulting event timeline with your on-call evidence.