Observability: logs, metrics and traces with OpenTelemetry and Prometheus

Generated by Claude through GigAI 0.1.11.10 on 2026-10-12.

Concept: Observability · also covers Logs, metrics and tracing, Telemetry · Step 28 of 35 in MLOps / ML platform engineer · version 1

Practice this

On this page

Observability is the ability to explain a system's internal state from the telemetry it emits, without shipping new code to ask a new question. For an ML platform this means metrics, logs and traces that are correlated with each other and extended with model-specific dimensions such as model version, queue depth and GPU use. OpenTelemetry standardizes how telemetry is produced and transported, and Prometheus is the common store and query engine for metrics.

Read more

How it is done in practice

A platform team ships observability as a default, not as a per-team project: a base image or library that configures the OpenTelemetry SDK, a Collector deployed as a DaemonSet plus gateway, and Prometheus scraping every pod that carries the right annotation or ServiceMonitor. Model servers expose a metrics endpoint out of the box, and a GPU exporter runs on every GPU node. Dashboards are generated from templates keyed on model name and version, and recording rules precompute the expensive latency and error ratios that alerts and SLO reports reuse. Cardinality is reviewed like any other cost, because one bad label can take down the metrics store.

Technologies

OpenTelemetryOpenTelemetry CollectorOTLPPrometheusPromQLAlertmanagerGrafanaNVIDIA DCGM ExporterJaegerGrafana TempoGrafana Loki

Diagram

flowchart LR
  A[Model server with OTel SDK] -->|OTLP| B[Collector agent per node]
  B --> C[Collector gateway]
  C -->|traces| D[Trace backend]
  C -->|logs| E[Log backend]
  A -->|metrics endpoint| F[Prometheus]
  G[GPU exporter] --> F
  F --> H[Alertmanager]
  F --> I[Dashboards]
  D --> I
  E --> I
flowchart LR
  A[Model server with OTel SDK] -->|OTLP| B[Collector agent per node]
  B --> C[Collector gateway]
  C -->|traces| D[Trace backend]
  C -->|logs| E[Log backend]
  A -->|metrics endpoint| F[Prometheus]
  G[GPU exporter] --> F
  F --> H[Alertmanager]
  F --> I[Dashboards]
  D --> I
  E --> I

Code

python
from prometheus_client import Counter, Histogram, start_http_server

REQS = Counter("inference_requests_total", "Requests",
               ["model", "version", "outcome"])
LAT = Histogram("inference_duration_seconds", "Latency",
                ["model", "version"],
                buckets=[.01, .025, .05, .1, .25, .5, 1, 2.5])

def predict(model, version, x, run):
    with LAT.labels(model, version).time():
        try:
            y = run(x)
            REQS.labels(model, version, "ok").inc()
            return y
        except Exception:
            REQS.labels(model, version, "error").inc()
            raise

start_http_server(9100)

Subtopics

Signals and how they correlate

Beyond three separate tools

OpenTelemetry defines signals: traces, metrics, logs and baggage. The value comes from correlating them. A resource describes the entity producing telemetry (service name, version, Kubernetes pod, cloud region) and is attached to every signal, so a latency spike in a metric can be narrowed to a pod, then to the traces from that pod, then to the log records that carry the same trace and span ids. For ML services, add the model name and model version as resource or span attributes so every signal can be sliced by the artifact that was actually serving.

Traces, context propagation and the Collector

Standard propagation

A trace is a tree of spans sharing a trace id. Context propagation carries that id across process boundaries, normally in request headers using the W3C Trace Context format, and through message metadata for queues. OpenTelemetry SDKs do this through propagators, and auto-instrumentation libraries cover common HTTP, gRPC and database clients. In an inference path, useful spans are: gateway, feature lookup, preprocessing, queue wait, model execution and postprocessing. Queue wait as its own span is what separates a slow model from an overloaded one.

The Collector as the pipeline

Applications export over OTLP to an OpenTelemetry Collector instead of directly to a vendor. The Collector runs receivers, processors and exporters: batching, attribute enrichment from Kubernetes metadata, redaction, and sampling. Tail-based sampling keeps all error and slow traces while dropping most healthy ones, which matters at inference volumes. Running an agent Collector per node plus a gateway Collector deployment is the usual topology, and it lets the backend change without touching application code.

Prometheus data model and instrumentation practice

How Prometheus works

Prometheus stores time series identified by a metric name and a set of key-value labels. It collects them by pulling over HTTP from targets it finds through service discovery or static configuration, stores them locally without depending on distributed storage, and exposes them through PromQL. Short-lived jobs that cannot be scraped push through an intermediary gateway. Alerting is evaluated in Prometheus and routed by Alertmanager. Each server is autonomous, which is why it remains usable during the outages you need it for.

What to instrument

The instrumentation guidance splits services by type. Online serving systems expose request count, error count and latency. Offline pipelines expose items in and out and time spent per stage. Batch jobs expose the timestamp of the last success, so the alert is on staleness. Use counters for things that only increase and compute rates at query time, gauges for values that go up and down, and histograms for latency so quantiles can be aggregated across replicas. Keep label cardinality bounded: never put user ids, request ids or raw prompts in labels.

Metrics specific to model serving

What generic service dashboards miss

A model server needs the standard request, error and duration metrics plus dimensions a web service lacks. Track queue depth and queue wait time, batch size distribution, and in-flight requests per replica. For GPUs, export utilization, memory used and temperature per device, mapped to pod and namespace. For LLM serving, track time to first token, tokens generated per second, and input and output token counts, because request latency alone depends on output length. Label by model name and version, not by request content, and record cost drivers such as GPU seconds per tenant so cost can be attributed.

Sources

Related lessons

Practice this
LLMOps: evals, tracing, guardrails and AI-assisted development with Claude Code and CursorModel monitoring: data quality, drift and alerting