Backend Development

Observability

Use logs, metrics, traces, and profiles to understand a running service without guessing.

What you will learn

By the end of this lesson, you will be able to:

  • choose the right signal among logs, metrics, traces, and profiles;
  • define useful request and dependency measurements;
  • avoid dangerous high-cardinality metric labels;
  • propagate correlation context across boundaries;
  • expose diagnostics without exposing them publicly.

Mental model: ask questions of a running system

Observability is the ability to understand internal behaviour from emitted signals. Each signal answers different questions.

SignalBest for
LogsDetailed events and failure context
MetricsTrends, rates, saturation, alerts, and service objectives
TracesThe path and timing of one request across components
ProfilesWhere CPU time, allocations, blocking, or contention occur

Do not force every detail into logs or every identifier into metrics.

Start with request metrics

For HTTP services, the RED model is a useful start:

  • Rate: requests per unit of time;
  • Errors: failed requests per unit of time;
  • Duration: request latency distribution.

Also observe dependency latency/errors and resource saturation such as connection-pool wait, queue depth, memory, CPU, and goroutine counts when they reflect actionable limits.

Control metric cardinality

Metric systems create a time series for every unique label combination. Unbounded values can create an expensive explosion.

Safe labels are usually from bounded sets:

  • route pattern, not raw URL path;
  • HTTP method;
  • status class such as 2xx or carefully bounded status code;
  • stable service or dependency name.

Unsafe labels commonly include:

  • user ID;
  • request ID;
  • email address;
  • raw URL containing resource IDs;
  • arbitrary error messages.

Put high-cardinality identifiers in logs or traces where appropriate and protected.

Correlate signals

A request ID links logs from one service. A trace ID links spans across services. Propagate supported tracing headers using a trusted instrumentation library rather than inventing a wire format.

Context carries request-scoped propagation information through function calls:

func (s *Service) Create(ctx context.Context, command CreateCommand) (Lesson, error) {
    return s.repository.Create(ctx, command)
}

Do not replace the request context with context.Background() inside the call chain; doing so discards cancellation, deadlines, and trace relationships.

Measure at boundaries

Useful instrumentation sits around operations:

started := time.Now()
lesson, err := repository.Find(ctx, id)
duration := time.Since(started)

metrics.ObserveRepositoryCall("lesson_find", duration, err)

The metric API should map errors to a bounded classification rather than using err.Error() as a label.

Avoid instrumentation that changes the behaviour being measured through blocking network calls on the request path. Exporters generally buffer or batch according to documented limits.

Profiles and diagnostics

The Go runtime can expose profiles understood by go tool pprof, including CPU and heap data. The net/http/pprof package can register diagnostic HTTP handlers.

Do not expose profiling endpoints to the public internet. Profiles and runtime diagnostics can reveal sensitive implementation and workload details, and profiling can add overhead. Put them on a protected administrative listener or collect profiles through an authenticated operational channel.

Measure before optimising. Start with a representative workload, capture evidence, change one relevant factor, and compare results.

Service-level thinking

An alert should connect to user impact and an action. “CPU is 80%” is not automatically a failure. “The proportion of successful requests under the latency target is falling” describes service behaviour more directly.

Define objectives based on what users need, then choose metrics that measure those outcomes.

Common mistakes

  • Using raw paths as labels: every resource ID can create a new time series.
  • Recording every event at error level: alerts become noise and real failures disappear.
  • Starting a new trace without preserving incoming context: distributed requests fragment.
  • Adding instrumentation after incidents only: missing baseline signals make diagnosis slow.
  • Making diagnostic endpoints public: internal state becomes externally accessible.
  • Optimising from one profile captured under an unrealistic workload: the result may not matter in production.

Practice

Design observability for POST /lessons. Specify one log event, request metrics, a trace span hierarchy, and a profiling strategy. Classify every attribute as bounded or unbounded.

Summary

  • Logs, metrics, traces, and profiles answer different questions.
  • Begin with request rate, errors, duration, and dependency saturation.
  • Keep metric labels bounded; never label metrics with user or request IDs.
  • Propagate context through every request-scoped boundary.
  • Protect runtime diagnostics and optimise only from representative evidence.

Official reference: Go diagnostics.

Ready to wrap up?

Mark this complete when you can explain the main idea and finish the practice task.

Copyright © 2026