Concurrent URL Checker

URL Checker — Cancellation and Tests

Prove worker limits, timeout behaviour, ordering, cleanup, and race-free execution with controlled servers.

What you will learn

By the end of this checkpoint, you will be able to:

  • test network behaviour with local controlled servers;
  • measure maximum in-flight requests safely;
  • coordinate cancellation tests without arbitrary sleeps;
  • verify deterministic ordering despite out-of-order completion;
  • use the race detector and goroutine checks responsibly.

Use httptest.Server

server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
    w.WriteHeader(http.StatusNoContent)
}))
t.Cleanup(server.Close)

The server provides a local URL and real HTTP transport behaviour without relying on the public internet.

Test output ordering

Create paths controlled by channels:

releaseFirst := make(chan struct{})

server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
    if r.URL.Path == "/first" {
        <-releaseFirst
    }
    w.WriteHeader(http.StatusOK)
}))
t.Cleanup(server.Close)

Start checking /first and /second, confirm the second handler is reached, then close releaseFirst. The first request finishes last, but returned results must follow input indexes.

Use channels to observe events. Avoid time.Sleep as the primary ordering mechanism.

Verify the worker bound

var current atomic.Int64
var maximum atomic.Int64
    release := make(chan struct{})

handler := http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
    active := current.Add(1)
    defer current.Add(-1)

    for {
        seen := maximum.Load()
        if active <= seen || maximum.CompareAndSwap(seen, active) {
            break
        }
    }

    <-release
    w.WriteHeader(http.StatusNoContent)
})

Start more jobs than workers. Wait until the expected number has entered, then assert maximum <= workers and release the handlers. The full test needs a bounded timeout so a broken pipeline fails instead of hanging forever.

Test request timeout

started := make(chan struct{})
server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
    close(started)
    <-r.Context().Done()
}))
t.Cleanup(server.Close)

Run a check with a short timeout, wait for started, and assert the result error matches context.DeadlineExceeded through errors.Is where the wrapping chain supports it.

Avoid comparing complete network error strings; their formatting is not a stable contract.

Test command cancellation

  1. Handler signals that a request began.
  2. Test cancels the parent context.
  3. Handler observes request-context cancellation.
  4. CheckAll returns within a bounded test deadline.
  5. Every worker completion signal is observed.

The test's outer deadline is a safety net, not the mechanism that creates the intended order.

Check response cleanup

A custom RoundTripper can return a response body whose Close method records the call:

type recordingBody struct {
    io.Reader
    closed atomic.Bool
}

func (b *recordingBody) Close() error {
    b.closed.Store(true)
    return nil
}

Inject an http.Client using the transport and assert the body is closed after check returns.

Race and leak validation

When a Go environment is available:

go test -race -count=20 ./...

Repeated tests increase the chance of exercising schedules but still do not prove correctness.

Goroutine-count assertions are noisy because the runtime and HTTP transport own goroutines. Prefer explicit worker completion channels or wait groups. A goroutine profile can help diagnose unexplained growth.

Final completion checklist

  • malformed URLs fail before workers start;
  • worker count and queue size are bounded;
  • every successful response body closes;
  • request and parent cancellation propagate;
  • output order matches input order;
  • per-result failure policy is tested;
  • public internet is not used by tests;
  • race-enabled repeated tests complete cleanly in a supported environment;
  • command help documents security scope and limits.

Practice

Add optional first-error cancellation. Ensure the collector continues draining until workers exit so a worker never blocks sending the error that triggered cancellation.

Summary

  • Local test servers make network behaviour controllable and repeatable.
  • Channels create deterministic test ordering without arbitrary sleeps.
  • Atomic counters can verify actual maximum concurrency.
  • Assert error identity or classification instead of unstable text.
  • Confirm owned goroutines finish; raw runtime goroutine counts are only diagnostic clues.

Ready to wrap up?

Mark this complete when you can explain the main idea and finish the practice task.

Copyright © 2026