Practice: Observability: logs, metrics and traces with OpenTelemetry and Prometheus

Generated by Claude through GigAI 0.1.11.10 on 2026-10-12.

Main tool: OpenTelemetry · OpenTelemetry is the one vendor-neutral API, SDK and protocol that produces metrics, logs and traces together with shared context, so every subtopic of this lesson can be practised with one tool while Prometheus and Jaeger act only as local backends. · Alternatives: Prometheus client libraries (metrics only, pull-based, no traces or logs), Micrometer (a JVM-only metrics facade), native Jaeger or Zipkin clients (tracing only, largely superseded by OTLP), eBPF auto-instrumentation (no code changes but less control over custom attributes), and proprietary vendor agents (quick to start but tied to one backend).

Every step links into the tool's own documentation; links were checked on 2026-10-12 · version 1

Setup (one time)

1. Install the prerequisites the getting-started guide lists, including the SDK for the sample application and a container runtime or local binaries for the backends.
Every later step builds on the sample application from this guide, so the toolchain has to work before any telemetry can be produced.
Explained in: Export to Prometheus and Grafana | OpenTelemetry (opentelemetry.io), checked 2026-10-12
Done when: The version command of the application SDK prints a version that meets the guide's minimum.
2. Install and start a local Prometheus with its OTLP receiver enabled, as the guide describes.
Prometheus is the local metrics store that the OpenTelemetry SDK pushes to over OTLP, so it must be listening before the first run.
Explained in: Export to Prometheus and Grafana | OpenTelemetry (opentelemetry.io), checked 2026-10-12
Done when: The Prometheus web interface opens in the browser on the local port and reports that it is ready.
3. Install and start a local Jaeger instance that accepts OTLP, as the guide describes.
Jaeger is the local trace backend for the tracing and context propagation steps, and it runs on a laptop in a single container.
Explained in: Export to Jaeger | OpenTelemetry (opentelemetry.io), checked 2026-10-12
Done when: The Jaeger web interface opens in the browser and shows an empty search page.

Steps

1. Create the sample application with OTLP export enabled and run it against the local Prometheus.
This is the shortest path from zero to telemetry leaving a process, and it shows the SDK wiring that every later step extends.
Explained in: Export to Prometheus and Grafana | OpenTelemetry (opentelemetry.io), checked 2026-10-12
Done when: The application runs without export errors on the console while it handles a few requests.
2. Query the sample application's metrics in the Prometheus web interface.
Seeing your own series in the store confirms the whole push path and shows how OpenTelemetry names and attributes arrive as Prometheus metric names and labels.
Explained in: Export to Prometheus and Grafana | OpenTelemetry (opentelemetry.io), checked 2026-10-12
Done when: A query for one of the application's metrics returns at least one series with a value that changes as you send requests.
3. Write a manual span around a simulated model inference call and add attributes such as model name and model version.
Spans and their attributes are the core objects of tracing, and model-specific attributes are what make a trace useful for an ML platform.
Explained in: Traces | OpenTelemetry (opentelemetry.io), checked 2026-10-12
Done when: Jaeger shows a trace for the application containing your named span with the model attributes on it.
4. Run a second small service that the first one calls over HTTP and enable context propagation between them.
Propagation is what turns separate spans from separate processes into one distributed trace, and it breaks silently when a hop drops the trace headers.
Explained in: Context propagation | OpenTelemetry (opentelemetry.io), checked 2026-10-12
Done when: Jaeger shows one trace with spans from both services under the same trace id, with the callee span as a child of the caller span.
5. Write a log record inside an active span using the OpenTelemetry logging integration and inspect its trace context fields.
Logs that carry the trace id and span id are how the three signals correlate, so you can jump from a log line to the exact request that produced it.
Explained in: Logs Data Model | OpenTelemetry (opentelemetry.io), checked 2026-10-12
Done when: The exported log record shows a trace id and span id that match a span you can find in Jaeger.
6. Create model serving instruments: a counter for inference requests, a histogram for inference latency, and gauges for queue depth and a simulated GPU utilization value (no GPU needed, emit a synthetic number).
Choosing the right instrument kind per measurement is the core of instrumentation practice, and these four are the serving metrics a platform team is asked for first.
Explained in: Metrics | OpenTelemetry (opentelemetry.io), checked 2026-10-12
Done when: Prometheus returns series for all four metrics, and the latency metric appears with bucket, sum and count series.
7. Configure a view that sets explicit latency buckets and drops a high-cardinality attribute such as request id, then set a cardinality limit.
Unbounded attribute values are the usual way a metrics pipeline falls over, and views let you fix that in the SDK without touching instrumentation code.
Explained in: Metrics | OpenTelemetry (opentelemetry.io), checked 2026-10-12
Done when: The latency metric in Prometheus shows your bucket boundaries and no longer carries the dropped attribute as a label.
8. Configure service name, service version and deployment environment as Resource attributes instead of repeating them as tags on every span.
Resource is how a build identifies itself across all signals, which is what lets you compare one released version of a model server against another.
Explained in: Best practices | OpenTelemetry (opentelemetry.io), checked 2026-10-12
Done when: Every span in Jaeger shows the same service name and version in its process or resource section without those values being set in span code.
9. Configure the metrics temporality preference for the OTLP exporter and compare cumulative with delta output.
Prometheus expects cumulative series, so knowing which temporality the exporter sends explains why counters and rates look right or wrong in the store.
Explained in: OTLP Exporter Configuration | OpenTelemetry (opentelemetry.io), checked 2026-10-12
Done when: With the cumulative setting, the request counter in Prometheus rises monotonically and a rate query over it gives a steady value under steady load.
10. Reproduce the whole export setup using only the standard OTLP environment variables, removing endpoints and protocols from application code.
Configuration through environment variables makes the same build run unchanged on a laptop, in CI and in a cluster, which is what makes the setup repeatable.
Explained in: OTLP metrics export to Prometheus | OpenTelemetry (opentelemetry.io), checked 2026-10-12
Done when: Starting the application from a clean shell with only the environment variables set still delivers metrics to Prometheus.
11. Start the OpenTelemetry demo with containers and open the Collector data flow dashboard to follow traces and metrics through receivers, processors and exporters.
The Collector is the hop that decouples applications from backends, and its own ingress and egress metrics are how you prove no telemetry is being dropped.
Explained in: Collector Data Flow Dashboard | OpenTelemetry (opentelemetry.io), checked 2026-10-12
Done when: The dashboard shows non-zero ingress and egress rates for both the traces pipeline and the metrics pipeline.
12. Apply the production usage guidance to the trace export setup and run the two services under steady load with those settings.
The defaults that are convenient for a first run are not the ones to operate with, and this step closes the gap between the laptop demo and a setup you could defend in review.
Explained in: Export to Jaeger | OpenTelemetry (opentelemetry.io), checked 2026-10-12
Done when: Traces from both services still arrive in Jaeger under load with the production settings on, and the application logs show no export errors.

Back to Observability: logs, metrics and traces with OpenTelemetry and Prometheus