Observability
What you will learn
By the end of this lesson, you will be able to:
- choose the right signal among logs, metrics, traces, and profiles;
- define useful request and dependency measurements;
- avoid dangerous high-cardinality metric labels;
- propagate correlation context across boundaries;
- expose diagnostics without exposing them publicly.
Mental model: ask questions of a running system
Observability is the ability to understand internal behaviour from emitted signals. Each signal answers different questions.
| Signal | Best for |
|---|---|
| Logs | Detailed events and failure context |
| Metrics | Trends, rates, saturation, alerts, and service objectives |
| Traces | The path and timing of one request across components |
| Profiles | Where CPU time, allocations, blocking, or contention occur |
Do not force every detail into logs or every identifier into metrics.
Start with request metrics
For HTTP services, the RED model is a useful start:
- Rate: requests per unit of time;
- Errors: failed requests per unit of time;
- Duration: request latency distribution.
Also observe dependency latency/errors and resource saturation such as connection-pool wait, queue depth, memory, CPU, and goroutine counts when they reflect actionable limits.
Control metric cardinality
Metric systems create a time series for every unique label combination. Unbounded values can create an expensive explosion.
Safe labels are usually from bounded sets:
- route pattern, not raw URL path;
- HTTP method;
- status class such as
2xxor carefully bounded status code; - stable service or dependency name.
Unsafe labels commonly include:
- user ID;
- request ID;
- email address;
- raw URL containing resource IDs;
- arbitrary error messages.
Put high-cardinality identifiers in logs or traces where appropriate and protected.
Correlate signals
A request ID links logs from one service. A trace ID links spans across services. Propagate supported tracing headers using a trusted instrumentation library rather than inventing a wire format.
Context carries request-scoped propagation information through function calls:
func (s *Service) Create(ctx context.Context, command CreateCommand) (Lesson, error) {
return s.repository.Create(ctx, command)
}
Do not replace the request context with context.Background() inside the call chain; doing so discards cancellation, deadlines, and trace relationships.
Measure at boundaries
Useful instrumentation sits around operations:
started := time.Now()
lesson, err := repository.Find(ctx, id)
duration := time.Since(started)
metrics.ObserveRepositoryCall("lesson_find", duration, err)
The metric API should map errors to a bounded classification rather than using err.Error() as a label.
Avoid instrumentation that changes the behaviour being measured through blocking network calls on the request path. Exporters generally buffer or batch according to documented limits.
Profiles and diagnostics
The Go runtime can expose profiles understood by go tool pprof, including CPU and heap data. The net/http/pprof package can register diagnostic HTTP handlers.
Measure before optimising. Start with a representative workload, capture evidence, change one relevant factor, and compare results.
Service-level thinking
An alert should connect to user impact and an action. “CPU is 80%” is not automatically a failure. “The proportion of successful requests under the latency target is falling” describes service behaviour more directly.
Define objectives based on what users need, then choose metrics that measure those outcomes.
Common mistakes
- Using raw paths as labels: every resource ID can create a new time series.
- Recording every event at error level: alerts become noise and real failures disappear.
- Starting a new trace without preserving incoming context: distributed requests fragment.
- Adding instrumentation after incidents only: missing baseline signals make diagnosis slow.
- Making diagnostic endpoints public: internal state becomes externally accessible.
- Optimising from one profile captured under an unrealistic workload: the result may not matter in production.
Practice
Design observability for POST /lessons. Specify one log event, request metrics, a trace span hierarchy, and a profiling strategy. Classify every attribute as bounded or unbounded.
Log event: lesson create completed
request_id unbounded → log only
lesson_id unbounded → log only when policy permits
status bounded
duration numeric value
Metrics:
http_requests_total{method="POST", route="/lessons", status_class="2xx"}
http_request_duration_seconds{method="POST", route="/lessons"}
repository_calls_total{operation="lesson_create", outcome="success"}
repository_duration_seconds{operation="lesson_create"}
Trace:
server span POST /lessons
child span lesson.Service.Create
child span database INSERT lesson
Profiles:
protected administrative endpoint or controlled on-demand capture
The exact metric and trace APIs depend on the chosen observability library. The design keeps label sets bounded and propagates the request context into service and repository calls.
Summary
- Logs, metrics, traces, and profiles answer different questions.
- Begin with request rate, errors, duration, and dependency saturation.
- Keep metric labels bounded; never label metrics with user or request IDs.
- Propagate context through every request-scoped boundary.
- Protect runtime diagnostics and optimise only from representative evidence.
Official reference: Go diagnostics.
Ready to wrap up?
Mark this complete when you can explain the main idea and finish the practice task.