Advanced Go

Advanced Concurrency

Design bounded pipelines with cancellation, backpressure, error propagation, and explicit ownership.

What you will learn

By the end of this lesson, you will be able to:

  • design a pipeline with clear channel ownership;
  • bound concurrency and queued work;
  • propagate cancellation and the first meaningful error;
  • distinguish fan-out, fan-in, semaphore, and worker-pool designs;
  • reason about goroutine leaks and partial results.

Begin with ownership

For every channel, document:

  • who creates it;
  • who sends;
  • who receives;
  • who closes it;
  • what cancellation means;
  • whether every sent value must be processed.

The goroutine that owns all sends is normally responsible for closing the channel. Receivers should not close a channel merely because they are finished receiving.

A pipeline stage

func square(ctx context.Context, input <-chan int) <-chan int {
    output := make(chan int)
    go func() {
        defer close(output)
        for {
            select {
            case <-ctx.Done():
                return
            case value, ok := <-input:
                if !ok {
                    return
                }
                select {
                case output <- value * value:
                case <-ctx.Done():
                    return
                }
            }
        }
    }()
    return output
}

Cancellation is checked while receiving and sending. Without the second check, the goroutine can leak when downstream stops reading.

Bound work with a fixed worker pool

func worker(ctx context.Context, jobs <-chan Job, results chan<- Result) error {
    for {
        select {
        case <-ctx.Done():
            return ctx.Err()
        case job, ok := <-jobs:
            if !ok {
                return nil
            }
            result, err := process(ctx, job)
            if err != nil {
                return err
            }
            select {
            case results <- result:
            case <-ctx.Done():
                return ctx.Err()
            }
        }
    }
}

Start a fixed number of workers. The worker count bounds simultaneous process calls. A bounded jobs channel limits queued work and creates backpressure.

Worker count should reflect the constrained resource. More workers can increase contention, database pool waits, rate-limit failures, or memory use.

Fan-out and fan-in

  • Fan-out: several workers receive from one input channel.
  • Fan-in: one coordinator combines several output channels into one.

When fanning in, the coordinator closes the combined output only after every forwarding goroutine finishes. A WaitGroup can coordinate those forwarders.

Error propagation

Decide whether:

  • the first error cancels all work;
  • errors are collected while independent work continues;
  • partial results are valid;
  • failed jobs are retried or returned to a durable queue.

Do not add an error channel without defining who drains it and who closes it. A single buffered result containing {Value, Err} is sometimes clearer.

The external golang.org/x/sync/errgroup package provides a useful pattern for related goroutines with error propagation and context cancellation. Understand its documented limits—such as whether goroutine count is bounded for your use—before adopting it.

Semaphores

A semaphore bounds entry to an operation while allowing goroutines to be started elsewhere. A worker pool bounds both worker goroutines and, with a bounded input, queued work.

Choose based on ownership:

  • fixed stream of jobs with backpressure → worker pool;
  • occasional independent operations sharing a scarce limit → semaphore may fit;
  • simple shared data invariant → mutex may be clearer than channels.

Backpressure

When producers are faster than consumers, the system must:

  • block producers;
  • buffer up to a fixed bound;
  • reject or shed work;
  • persist work durably;
  • scale consumers within safe limits.

An unbounded in-memory queue only delays failure while memory grows.

Cancellation is not cleanup completion

Calling cancel() requests that work stop. The caller must still wait for owned goroutines to return before closing dependencies they use.

Design blocking library calls with contexts or other interruption mechanisms. A goroutine cannot be safely killed from outside.

Common mistakes

  • Starting one goroutine per item without a bound: a traffic spike becomes a resource spike.
  • Returning early while upstream still sends: upstream goroutines block forever.
  • Closing a shared input from one receiver: other senders panic.
  • Using a large buffer to hide deadlock: ownership remains incorrect.
  • Assuming scheduler fairness: correctness must not depend on a goroutine receiving an equal turn within a precise time.
  • Cancelling but not waiting: dependencies close while goroutines still use them.

Practice

Build a bounded URL-check pipeline with five workers, at most twenty queued jobs, per-request timeout, first-error cancellation option, ordered result reconstruction, and complete goroutine cleanup.

Summary

  • Document channel creation, sending, receiving, closing, cancellation, and delivery semantics.
  • Bound simultaneous work and queued work separately.
  • Check cancellation around both receives and sends in pipelines.
  • Decide how errors and partial results affect sibling work.
  • Cancellation requests shutdown; waiting confirms completion.

Official reference: Go Concurrency Patterns: Pipelines and cancellation.

Ready to wrap up?

Mark this complete when you can explain the main idea and finish the practice task.

Copyright © 2026