Limit Metric Cardinality in LLM API Gateways

Last reviewed: October 29, 2026 (Asia/Shanghai)

Direct answer

Control LLM API metric cardinality by allowing a small, defined set of label keys and values before recording measurements. Budget the complete combinations across replicas and histogram components. Keep request identifiers in controlled correlation records, and treat SDK overflow as a loss of diagnostic detail that needs attention.

A gateway that adds request_id to its latency histogram can create a fresh group of series for every call. Switching to a hashed request ID preserves that growth. Even individually reasonable labels such as route, model, provider, region, outcome, and customer can multiply into an expensive metric when combined.

Prometheus’s naming guidance states that each unique label combination creates a time series and warns against unbounded values such as user IDs. Its instrumentation guidance connects those series to RAM, CPU, disk, and network costs. For an LLM gateway, the operational risk includes losing visibility while retries or provider failures are already increasing traffic.

Who this is for

This guide is for platform engineers and SREs operating LLM gateways with Prometheus-compatible monitoring, OpenTelemetry metrics, or both. It assumes you control the application instrumentation and can inspect the exported data.

The policy, calculations, and acceptance checks below are recommendations derived from the official references. They are not reported deployment results. The proposed label names and enums are application conventions; verify their implementation in your SDK and exporter.

Key takeaways

• Bound label values as well as label keys. An allowed model key can still contain an unlimited set of names.
• Calculate combinations across the fleet, including histogram representation and resource labels.
• Apply normalization before recording measurements; use SDK Views as an additional control.
• Keep a lean user-outcome metric separate from provider-attempt diagnostics.
• Inspect overflow, missing series, and exemplars during verification.
• Review instrumentation changes as changes to an operational dependency.

Sources checked

Four primary public pages were checked on October 29, 2026 in Asia/Shanghai, equivalent to October 28 UTC:

• Prometheus: Metric and label naming supports the relationship between label combinations and series, the warning against unbounded identifiers, and consistent metric units.
• Prometheus: Instrumentation explains label resource costs, stable metric names, failure and attempt counters, and missing-metric handling.
• OpenTelemetry: Metrics SDK defines View attribute filtering, cardinality-limit configuration, overflow aggregation, and the possibility that filtered attributes remain in exemplars.
• Prometheus: Histograms and summaries explains the separate component series of classic histograms and the composite samples of native histograms.

These are maintained technical references for an evergreen engineering task. Their relevance comes from the documented data model and SDK behavior, rather than a recent product announcement. SDK support and exporter translation still need verification against the versions you deploy.

Contract details to verify

Give each dimension a finite vocabulary

Start with the questions an on-call engineer must answer: which operation is failing, which provider pool is affected, and what happened? Add a dimension only when its diagnostic value justifies its complete combination count.

A starting policy could be:

A key allowlist alone cannot enforce this policy. Validate values too, including values supplied by plugins, customers, and newly discovered model catalogs. Treat generated model output as untrusted telemetry input as well. Map unrecognized values to a constant such as other; never construct other_<original_value>.

Hashing or truncating strings does not establish a useful cardinality bound. A hash normally preserves distinct identifiers, while truncation still permits many different values. Limit instrument names too: generating a separate metric name for each model or customer merely moves the unbounded dimension.

Keep detailed identifiers under the trace correlation contract . Apply retention and redaction rules there, and exclude customer content from metric attributes entirely.

Budget combinations and histogram components

For one metric, multiply the allowed value counts to estimate a maximum combination count. Then include independently labeled targets and the exported representation. The product is an upper bound when some combinations cannot occur; measure the populated combinations as well.

For example, suppose a diagnostic metric has three route classes, two provider pools, and seven outcomes:

maximum attribute combinations per target = 3 × 2 × 7 = 42
classic histogram component series per target = 42 × (10 + 2) = 504
component series across 12 distinct targets = 504 × 12 = 6,048

Here, ten means ten exported bucket series, including the +Inf bucket; two adds _sum and _count. The calculation covers those components only. Inspect the actual exporter for additional series and resource-derived labels. The numbers are illustrative, not a recommended budget.

The Prometheus histogram reference documents this component representation. A native histogram stores a composite histogram sample in one series per label combination. That changes the bucket-series multiplier, but each new identifier still creates another label combination, and the histogram samples still consume resources.

Set separate budgets for individual instruments and the whole service. An SDK limit on one instrument is not a fleet-wide allowance. Include simultaneous old and new replicas during a rollout, and measure label churn over time: repeatedly replacing model or deployment values can accumulate historical series even when the current set looks small.

Enforce the policy before aggregation

Normalize raw gateway fields before calling the metrics API. Use a fixed mapping for routes, provider pools, model families, and outcome classes. A small policy-violation counter can identify which approved field failed validation without putting the rejected value into another label.

The OpenTelemetry Metrics SDK specification provides an attribute_keys View setting that retains selected keys and ignores the others. It also describes aggregation_cardinality_limit and recommends enforcing cardinality after attribute filtering. Confirm that the deployed language SDK supports those settings and that the View actually matches the instrument.

The specification recommends a default limit of 2,000 when no more specific configuration applies. Treat that as a specification default to verify, rather than an approved operational budget. Several instruments, readers, resources, or processes can each contribute their own data.

Inspect every emitted stream. An additional unfiltered View or exporter path can preserve the original problem. Downstream filtering can reduce what reaches storage, but it cannot undo aggregation work that already happened inside the application. Application normalization and SDK filtering therefore protect different points in the pipeline.

Preserve totals while exposing lost detail

OpenTelemetry defines an overflow attribute set containing only otel.metric.overflow=true. Measurements that cannot receive their own aggregation because of the limit contribute to that synthetic aggregation.

For synchronous instruments, the specification requires each measurement to be reflected exactly once in either its ordinary aggregation or overflow, without double counting or dropping it during overflow. Verify this behavior in your implementation and end-to-end export path. The specification’s aggregation guarantee does not prove that every later network export succeeds.

Overflow loses the original diagnostic dimensions. A total across normal and overflow points can remain useful, while a provider-filtered chart misses the overflow contribution because it has no provider label. Excluding overflow and calling the remainder a complete total creates a misleading denominator.

Maintain a lean final-user-outcome counter and separate attempt diagnostics. One action that succeeds after two upstream attempts should produce one final action result and two attempt results. Use a smaller dimension set for the user-facing indicator so a diagnostic expansion has less opportunity to compromise it. This complements the error-budget alerting guidance .

Make overflow and telemetry freshness visible. A disappeared route series is not evidence that the route recovered. Check expected coverage and the exporter’s own health before using the data to change provider routing.

Inspect exemplars separately

The Metrics SDK reference explicitly warns that an attribute removed by a View may still be exported as a filtered attribute on an exemplar. Filtering a label out of the metric stream therefore does not prove the value disappeared from the entire payload.

Keep prompts, outputs, credentials, and user identifiers out of metric measurements at their origin. Where exemplars are enabled, inspect their attributes and correlation policy. The specification describes disabling exemplars with the AlwaysOff filter or using custom filtering and reservoir behavior. Verify the actual SDK configuration before relying on it for sensitive-field removal.

Failure modes

Every model alias becomes a label value. Catalog discovery expands the vocabulary without an instrumentation review. Use the approved family map and one fallback value; update the map deliberately.

A resource attribute reintroduces tenant identity. Removing a measurement label leaves the same identifier elsewhere in the exported identity. Inspect resource attributes and their translation into backend labels as part of the full series budget.

A histogram looks cheap because only its name was counted. Count the component series actually emitted by classic histograms, or the native representation actually ingested. Do not assume exporter conversion from the instrument’s name.

Overflow makes a failing provider disappear. The synthetic point has no original provider dimension. Show that diagnostic coverage is degraded, include overflow in applicable totals, and investigate the instrumentation limit.

An incident response adds a raw error label. The busiest failure path creates the most new values. Use bounded classes and controlled trace inspection. Hold or revert the instrumentation change if it threatens monitoring, while checking service health through independent signals.

A rollout temporarily doubles the budget. Old and new label schemas coexist, and retained history persists after rollback. Plan that overlap and compare schema populations without double counting the same logical events.

Verification before rollout

Use synthetic gateway events and a local exporter first. These are acceptance criteria, not tests reported as executed for this article.

Inspect both SDK output and backend ingestion. Record SDK and exporter versions, temporality, View selection, configured limits, populated combinations, overflow, scrape or export failures, and collection freshness. Measure process memory and export latency under representative event volume before assigning a production budget.

FAQ

Is hashing tenant IDs enough?

No. It changes the representation while usually retaining the number of distinct tenants. Use a finite approved cohort if the metric needs customer segmentation, and keep individual investigation in a controlled system.

Does the SDK limit protect the whole monitoring service?

No. Its scope is the relevant aggregation, and multiple instruments and processes contribute data. Verify SDK behavior, then budget the complete exported fleet and retained series separately.

Should we raise the limit whenever overflow appears?

First inspect the dimension that grew, the expected series count, and whether an unfiltered stream escaped the policy. Raise a limit only after capacity review and a clear diagnostic need. Overflow should prompt investigation rather than an automatic increase.

Can filtering labels remove sensitive data completely?

A View alone cannot establish that. OpenTelemetry warns that filtered attributes may remain in exemplars, and resource attributes have separate handling. Prevent sensitive values from entering metric measurements and inspect the complete export.

Reader next step

Choose one gateway latency histogram and inventory its keys, actual values, exporter components, and resource labels. Calculate its fleet-wide upper bound, define a finite value map, and add the unique-ID and forced-overflow checks. Approve a rollout allowance and an owner for overflow response before expanding the policy to the rest of the gateway.