Practice: Observability: logs, metrics and agent traces
Generated by Claude through GigAI 0.1.11.10 on 2026-10-12.
Main tool: OpenTelemetry · OpenTelemetry is the vendor-neutral standard that defines logs, metrics, traces, context propagation and the GenAI span conventions in one place, so what you practise here carries over to whichever backend a customer already runs. · Alternatives: Prometheus with Grafana covers metrics and dashboards only; Jaeger stores and displays traces but does not instrument code; Datadog is a paid hosted suite with its own agent; Langfuse, LangSmith and Arize Phoenix are LLM-focused trace and evaluation tools that view model calls and prompts rather than the whole system.
Every step links into the tool's own documentation; links were checked on 2026-10-12 · version 1
Setup (one time)
1. Read the overview of the main OpenTelemetry components so you know what the API, SDK, OTLP and the Collector each do before installing anything.
Every later step names one of these parts. Knowing which piece creates telemetry and which piece ships it stops you from debugging the wrong layer.
Explained in: What is OpenTelemetry? | OpenTelemetry (opentelemetry.io), checked 2026-10-12
Done when: You can say in one sentence each what the API, the SDK, OTLP and the Collector are responsible for.
2. Install the prerequisites listed for the Prometheus and Grafana export walkthrough, including the .NET SDK, on your laptop.
The official walkthroughs in this path use a small .NET application as the telemetry source. Everything runs locally with no paid cloud and no GPU.
Explained in: Export to Prometheus and Grafana | OpenTelemetry (opentelemetry.io), checked 2026-10-12
Done when: The dotnet command reports an installed SDK version in your terminal.
3. Install and start Prometheus locally with its OTLP receiver enabled, as the walkthrough describes.
You need a local metrics backend that accepts OTLP so the first run produces something you can query.
Explained in: Export to Prometheus and Grafana | OpenTelemetry (opentelemetry.io), checked 2026-10-12
Done when: The Prometheus web interface opens in your browser on the local port.
4. Install and start Grafana locally and add Prometheus as a data source.
Dashboards and alert thresholds are built in Grafana in later steps, so the connection to Prometheus must already work.
Explained in: Export to Prometheus and Grafana | OpenTelemetry (opentelemetry.io), checked 2026-10-12
Done when: Grafana reports that the Prometheus data source was saved and tested successfully.
Steps
1. Create the sample .NET application with OTLP export and run it so it emits metrics to your local Prometheus.
This is the shortest path to real telemetry leaving a process over OTLP. It gives you a working baseline to change in every later step.
Explained in: Export to Prometheus and Grafana | OpenTelemetry (opentelemetry.io), checked 2026-10-12
Done when: The application runs without export errors in its console output.
2. Query the application metrics in the Prometheus interface and confirm the values change as the application runs.
Seeing the metric arrive proves the whole export path works. It also shows you how instrument names appear on the backend side.
Explained in: Export to Prometheus and Grafana | OpenTelemetry (opentelemetry.io), checked 2026-10-12
Done when: A query for the application's metric returns a series with recent, changing samples.
3. Create a Grafana dashboard with one panel for request rate and one for latency from the sample application.
A dashboard is where the metrics signal becomes something an on-call person reads. Building one makes you choose which numbers matter.
Explained in: Export to Prometheus and Grafana | OpenTelemetry (opentelemetry.io), checked 2026-10-12
Done when: The dashboard shows two panels with live data from the running application.
4. Write one service level indicator and one service level objective for the sample application, and note which metric would trigger an alert when the objective is at risk.
Alerts should fire on user-visible reliability, not on every noisy number. The primer explains how indicators and objectives connect metrics to that decision.
Explained in: Observability primer | OpenTelemetry (opentelemetry.io), checked 2026-10-12
Done when: You have a written indicator, an objective with a target value, and one named alert condition tied to a metric on your dashboard.
5. Read the logs section of the primer and compare what a log line, a metric sample and a span each tell you about the same request.
The three signals answer different questions. Knowing which one to reach for is the basis for deciding what to alert on and what to keep only for investigation.
Explained in: Observability primer | OpenTelemetry (opentelemetry.io), checked 2026-10-12
Done when: You can state for one failing request which signal detects it, which locates it and which explains it.
6. Create a parent span with two child spans in the sample application, add attributes and a span event, set an error status on one child, and export them to the console.
Spans, attributes, events and status are the core object model of tracing. Agent traces later in this path are built from exactly these parts.
Explained in: Traces | OpenTelemetry (opentelemetry.io), checked 2026-10-12
Done when: The console output shows three spans sharing one trace id, with parent ids, your attributes and one error status.
7. Run a second small service, call it from the first over HTTP, and configure context propagation so both services write spans into the same trace.
Distributed tracing only works when the trace context crosses the process boundary. Doing it by hand once shows what the propagation headers carry.
Explained in: Context propagation | OpenTelemetry (opentelemetry.io), checked 2026-10-12
Done when: Spans from both services show the same trace id and the second service's span lists the first service's span as its parent.
8. Write an agent trace by hand: one span for the agent task, child spans for a model call and a tool execution, with names and attributes taken from the generative AI semantic conventions. Use a stubbed model response so no paid API is needed.
Shared attribute names for model, token counts and tool names are what let any backend explain cost and latency per task. A stub keeps the exercise on your laptop.
Explained in: Semantic Conventions | OpenTelemetry (opentelemetry.io), checked 2026-10-12
Done when: The exported trace shows an agent span with model and tool child spans carrying convention-named attributes for the model and token usage.
9. Configure a local Collector pipeline that deletes or redacts the attributes holding prompt and completion text before the spans are exported.
Prompt content often contains personal or confidential data and should not reach a shared backend by default. Removing it in the pipeline protects every service at once.
Explained in: Transforming telemetry | OpenTelemetry (opentelemetry.io), checked 2026-10-12
Done when: Spans leaving the Collector no longer contain the prompt text attribute while token counts and model name remain.
10. Configure sampling in the local Collector so that only a fraction of successful agent traces is kept while traces with errors are always kept.
Agent traces are large and telemetry volume drives cost. Sampling is the main control for keeping the useful traces and dropping the routine ones.
Explained in: Agent-to-gateway deployment pattern | OpenTelemetry (opentelemetry.io), checked 2026-10-12
Done when: After many requests the backend holds all error traces and visibly fewer successful ones than were sent.
11. Write an alert rule for your service level objective and a matching runbook entry, modelled on the published alert runbooks, and keep both in version control next to the application.
An alert without a runbook leaves the responder guessing. Storing rules and runbooks as files makes the alerting setup reproducible on any machine.
Explained in: Prometheus Alerts Runbooks | OpenTelemetry (opentelemetry.io), checked 2026-10-12
Done when: The repository contains one alert rule file and one runbook entry stating meaning, impact, diagnosis and mitigation.
12. Start Jaeger locally, point the trace exporter at it over OTLP, and open the agent trace from the earlier step in the Jaeger interface.
A trace backend turns span output into a timeline you can search by service and duration. This is how you explain a slow or expensive agent task to a customer.
Explained in: Export to Jaeger | OpenTelemetry (opentelemetry.io), checked 2026-10-12
Done when: Jaeger shows the agent trace as a timeline with the model call and tool spans nested under the agent span across both services.
13. Configure the application for the production usage guidance: a named service resource, an OTLP endpoint taken from configuration instead of code, and export through the Collector with redaction and sampling in place.
Production differs from the demo in where the endpoint lives, how the service is identified and what is filtered before storage. Applying the guidance last ties every earlier step into one deployable setup.
Explained in: Export to Jaeger | OpenTelemetry (opentelemetry.io), checked 2026-10-12
Done when: Changing only configuration switches the export target, and traces in Jaeger arrive under your service name with prompt content removed.