Expose the Tail Latency Your LLM API Load Test Is Hiding

Last reviewed: 2026-09-12

Direct answer

An LLM API load test can hide tail latency when each new request waits for a previous request to finish. As the API slows, the load generator also slows, fewer requests reach the overloaded system, and the measured latency can look healthier precisely when production behavior would be getting worse. That measurement failure is coordinated omission.

For LLM API coordinated omission load testing, schedule arrivals independently of request completion whenever real users or upstream jobs can continue to arrive during a slowdown. Record the intended schedule, actual request starts, generator-side drops, service outcomes, and latency distributions separately. A test has not demonstrated performance at 20 arrivals per second merely because it completed some requests successfully; it must also show that it continued offering approximately that workload throughout slow periods.

The Grafana k6 explanation of open and closed models describes the central distinction. A closed model starts another iteration after the previous one finishes, so response time influences the next arrival. An open model schedules arrivals independently of iteration completion. The Red Hat performance team’s coordinated omission analysis frames the missing measurement as wait time: a blocked generator can omit work that would have joined the queue in a real open system.

Open arrival is not a universal replacement for fixed concurrency. A bounded worker queue, a single sequential agent session, or a user journey with dependent steps can legitimately behave as a closed loop. The test model must match the production question. For an interactive workflow, a useful hybrid may allow new users to arrive independently while preserving sequential dependencies inside each user’s session. What matters is that service slowdown must not silently change the workload dimension you claim to be holding constant.

For streaming LLM calls, do not reduce the result to one response-time percentile. Capture time to first token, end-to-end latency, and inter-token latency using frozen definitions. Also report the scheduled arrival rate, achieved start rate, completion rate, errors, timeouts, and generator drops. This makes it possible to distinguish an API that queued work, an API that generated slowly, and a load generator that failed to deliver the test.

Who this is for

This guide is for SREs, platform engineers, performance engineers, and reliability owners who test chat, completion, or other generative routes. It is especially relevant when requests stream for different lengths of time, dynamic queueing is visible in time to first token, or a fixed-concurrency benchmark is being used to justify an arrival-rate SLO.

It assumes the operator can run a bounded test in an approved environment and obtain both load-generator and gateway telemetry. It does not assume a particular provider or test tool.

Key takeaways

Choose open, closed, or hybrid arrivals from the behavior being modeled, not from a convenient tool default.
Treat scheduled arrivals, actual starts, completions, and generator drops as different counters.
Keep failures and timeouts in the outcome ledger; a successful-response-only histogram is not a reliability result.
Measure TTFT, end-to-end latency, and ITL separately for streaming requests.
Evaluate short time windows as well as whole-run percentiles so a temporary stall cannot disappear inside a long aggregate.
Mark a run inconclusive when the generator cannot sustain the declared schedule.

Sources checked

Grafana k6 documentation on open and closed models explains that closed-model throughput falls as iterations take longer and that arrival-rate executors decouple new starts from response duration.
Red Hat’s coordinated omission methodology explains how unintended backpressure on a load driver omits waiting work and separates wait time from service time.
Artillery’s workload-model analysis describes coordinated omission when an open system is tested by a closed generator and discusses hybrid models with open user arrivals and dependent actions inside each user flow.
NVIDIA’s LLM benchmarking metric definitions defines TTFT, end-to-end request latency, ITL, system token throughput, and completed-request throughput. It also cautions that tool implementations vary, so definitions must align before results are compared.

These sources were publicly reachable when reviewed. Their claims are used for workload-model and metric definitions; the pass criteria below remain an operator-owned contract for the system being tested.

Contract details to verify

Start by writing the exact claim the run should test. For example: the route should accept a specified arrival profile for a defined plateau while remaining within agreed TTFT, end-to-end latency, error, and generator-drop limits. If the question is instead maximum throughput at a fixed concurrency, say so explicitly. Those are different experiments and should not share a pass label.

Freeze these inputs before comparing runs:

Arrival model, target rate, ramp shape, plateau duration, and drain period.
Streaming mode and the event that counts as first content and final content.
Prompt-shape distribution, input-length bands, requested output limits, and representative output-length bands.
Route and model class, gateway policy version, region, timeout policy, and retry behavior.
Warm-up treatment and the time windows excluded from the steady-state comparison.
Safety ceilings for in-flight work, spend, queue depth, and test duration.

For dependent user flows, document which actions must wait and which new users can arrive independently. Artillery’s analysis is useful here: an open outer arrival loop can coexist with closed sequential steps inside a session. Do not flatten a transactional flow into constant independent requests if doing so changes the system behavior under test.

Maintain mutually exclusive counters. After the final drain, the accounting should reconcile:

scheduled = started + generator_dropped
started = succeeded + rejected + timed_out + transport_failed + canceled

During the run, add an in-flight term to the second line. If the counters do not reconcile, do not infer missing outcomes from the latency histogram. Repair the instrumentation and rerun.

Use distinct timing boundaries:

Scheduler lag: planned start to actual send attempt.
TTFT: send attempt to the first response event containing output content.
End-to-end latency: send attempt to final output content.
ITL: the chosen average or distribution of intervals between generated content tokens, with the treatment of the first token stated explicitly.
User-observed first-content time: scheduler lag plus TTFT, when generator-side waiting represents waiting a production caller would experience.

NVIDIA defines TTFT from query submission to the first received token and notes that it generally includes API queueing, prefill, and network latency. Its end-to-end definition runs from submission through the final token, while its ITL definition excludes TTFT. Other tools may calculate ITL differently, so store the definition beside the result rather than relying on the metric name alone.

A sanitized per-request record can look like this:

run_id: loadtest-042
request_id: req-1042
window_id: window-06
route_id: stream-a
model_class: test-model-a
prompt_shape_id: shape-07
scheduled_offset_ms: 12000
send_offset_ms: 12004
first_content_offset_ms: 12380
final_content_offset_ms: 14120
scheduler_lag_ms: 4
input_token_count: 900
output_token_count: 240
http_status: 200
outcome: success
retry_count: 0
generator_dropped: false

Keep prompt text, response text, customer identifiers, request headers, and authentication material out of this record. A shape identifier and length bands are normally enough to group performance without copying content into reliability logs.

Happy-path operator workflow

Confirm the route, model class, workload fixture, retry setting, safety limits, and metric definitions. Run a small functional check before generating load.
Verify that the generator has sufficient workers, connections, CPU, and network headroom for the target arrival profile. Synchronize its clock source and start generator health telemetry.
Begin the bounded ramp, then hold the declared arrival plateau. Watch scheduled starts, actual starts, scheduler lag, generator drops, in-flight work, rejections, and timeouts in short windows.
Let the final requests drain within the declared drain limit. Reconcile all counters before computing the final report.
Compare steady-state TTFT, end-to-end latency, ITL, outcomes, and achieved start rate with the prewritten thresholds. Preserve both whole-run and windowed distributions.
Call the run a pass only when the workload was actually delivered and every required reliability threshold passed.

Error-path operator workflow

If scheduler lag rises or generator drops appear while the generator is resource-bound, stop claiming that the target rate was tested. Preserve the partial run as inconclusive, increase generator capacity or distribute the load safely, and repeat the same contract.
If the generator remains healthy but the API rejects, times out, or queues requests beyond a safety ceiling, record a system-under-test failure at that workload. Do not delete those outcomes or replace them with successful retries.
If an abort ceiling is reached, stop the run safely, retain the last complete windows, and label the result as an aborted failure rather than a completed capacity measurement.
If counters do not reconcile or content-event parsing fails, classify the run as an instrumentation failure. Fix the measurement path before changing the service.

If the test reveals a queue that grows as arrivals remain steady, follow up with the site’s guide to adaptive concurrency limits for LLM API routes . The load test should expose that condition; the gateway control should contain it.

Failure modes

A fixed worker count self-throttles. Each worker waits for a long stream to finish before starting another request. The test therefore sends less traffic during the slowdown and reports latency from a smaller-than-declared offered load.

Generator exhaustion is mistaken for API capacity. An open scheduler intends to start work, but its worker pool, connection pool, CPU, or network path is exhausted. If dropped starts and scheduler lag are not recorded, the report can resemble coordinated omission even though an open-arrival executor was selected.

Only successful requests enter the histogram. Rejections, transport failures, cancellations, and timeouts disappear while the remaining successful sample looks fast. Keep a complete outcome ledger and report latency alongside outcome rates.

Completion throughput is presented as arrival rate. NVIDIA defines requests per second from successfully completed requests over the benchmark interval. That is useful, but it is not the same as scheduled or achieved request starts. Report all three when validating an offered-load claim.

One aggregate smooths over a stall. A short queueing event can be diluted by many normal observations before and after it. Examine fixed windows and preserve the time series for starts, in-flight work, TTFT, errors, and generator health.

Metric names hide incompatible definitions. One tool may include the first-token interval in ITL while another excludes it. Streaming parsers may also disagree about empty terminal events. Freeze the event boundaries and formulas before comparing providers, models, or releases.

The prompt and output mix drifts. A run with shorter prompts or outputs can look better without any infrastructure improvement. Compare matched distributions and report the length bands next to latency results.

Retries create an undeclared workload. An automatic client retry can increase attempts while the report counts only logical requests. Either disable retries for the capacity experiment or log every attempt and specify whether the arrival contract applies to user actions or attempts.

Timeouts censor slow completions. A short client timeout can make the measured successful-response tail look smaller by removing the slowest calls. Count timeouts as outcomes and publish the timeout boundary. For route-specific design, review streaming and non-streaming timeout checks .

FAQ

Is a fixed-concurrency LLM benchmark always wrong?

No. It can answer questions about throughput or latency at a fixed number of in-flight requests, and it can represent genuinely closed workflows. It becomes misleading when its results are used to claim behavior at a fixed external arrival rate that the test did not maintain.

Does an open-arrival test guarantee valid results?

No. The scheduler can still run out of workers or other generator resources. Open scheduling must be paired with scheduled-versus-started accounting, scheduler-lag measurements, generator-drop counters, and generator health telemetry.

Which latency should be the primary LLM SLO?

That depends on the user experience. TTFT captures the wait before visible output begins, end-to-end latency captures the full response, and ITL characterizes generation cadence after the first token. A streaming feature often needs more than one threshold, plus an outcome-rate objective.

Can coordinated omission be corrected after the run?

Not reliably from completed-request latency alone when the missing work was never started. The Red Hat analysis shows why omitted waiting work changes the population being measured. Preserve any generator backpressure evidence, correct the workload model or capacity, and rerun the same contract.

Should generator-side waiting be included in API latency?

Keep it separate first. API TTFT should retain its documented send boundary, while scheduler lag shows whether the generator delivered work on time. If the wait represents a queue a real caller would have encountered, also publish a user-observed measure that includes it. Separate components make the diagnosis possible; the combined measure protects the experience claim.

How should a temporary overload be reported?

Show windowed scheduled starts, actual starts, in-flight work, outcomes, TTFT, and end-to-end latency around the event. State whether the generator maintained the schedule. Do not summarize the event with an average alone.

Reader next step

Take one existing fixed-concurrency benchmark and write down the claim it is supposed to prove. If that claim names an arrival rate, rerun the same prompt and output mix with an open or hybrid arrival model. Chart scheduled starts, actual starts, completions, generator drops, TTFT, and end-to-end latency in the same short windows.

Before accepting the result, reconcile the counters and inspect the worst window. If arrivals held and the API degraded, you found a service capacity boundary. If arrivals fell because the generator blocked, you found a measurement boundary. Address that boundary, repeat the unchanged contract, and then use overload signal triage for LLM API on-call to turn the observed queueing and rejection signals into an operational response.