Concurrent URL Checker

URL Checker — Worker Pipeline

Implement bounded workers, request lifetimes, result ownership, and ordered collection.

What you will learn

By the end of this checkpoint, you will be able to:

  • start exactly the configured number of workers;
  • stop sends and receives when the context is cancelled;
  • close result channels only after every worker returns;
  • release HTTP response bodies correctly;
  • reconstruct deterministic output from nondeterministic completion.

Check one URL

func check(ctx context.Context, client *http.Client, job Job, timeout time.Duration) Result {
    started := time.Now()
    result := Result{Index: job.Index, URL: job.URL.String()}

    requestCtx, cancel := context.WithTimeout(ctx, timeout)
    defer cancel()

    request, err := http.NewRequestWithContext(requestCtx, http.MethodGet, result.URL, nil)
    if err != nil {
        result.Err = fmt.Errorf("create request: %w", err)
        return result
    }

    response, err := client.Do(request)
    result.Duration = time.Since(started)
    if err != nil {
        result.Err = err
        return result
    }
    defer response.Body.Close()

    result.Status = response.StatusCode
    _, copyErr := io.Copy(io.Discard, io.LimitReader(response.Body, 4<<10))
    if copyErr != nil {
        result.Err = fmt.Errorf("read response body: %w", copyErr)
    }
    return result
}

Reading only 4 KiB prevents unbounded download. Because the body may contain more data, the connection may not be reusable in every case. For a checker, correctness and bounds are more important than forcing reuse of arbitrarily large bodies. A specialised transport or HEAD policy can be evaluated separately.

Redirects follow the client's policy and can change the destination. Configure CheckRedirect when redirect count or destination restrictions matter.

Worker loop

func worker(
    ctx context.Context,
    client *http.Client,
    jobs <-chan Job,
    results chan<- Result,
    timeout time.Duration,
) {
    for {
        select {
        case <-ctx.Done():
            return
        case job, ok := <-jobs:
            if !ok {
                return
            }

            result := check(ctx, client, job, timeout)
            select {
            case results <- result:
            case <-ctx.Done():
                return
            }
        }
    }
}

Cancellation is observed around both job receipt and result delivery.

Coordinator owns channel lifecycle

func CheckAll(ctx context.Context, client *http.Client, jobs []Job, config Config) ([]Result, error) {
    jobChannel := make(chan Job, min(len(jobs), config.Workers*2))
    resultChannel := make(chan Result, config.Workers)

    var workers sync.WaitGroup
    workers.Add(config.Workers)
    for range config.Workers {
        go func() {
            defer workers.Done()
            worker(ctx, client, jobChannel, resultChannel, config.RequestTimeout)
        }()
    }

    go func() {
        defer close(jobChannel)
        for _, job := range jobs {
            select {
            case jobChannel <- job:
            case <-ctx.Done():
                return
            }
        }
    }()

    go func() {
        workers.Wait()
        close(resultChannel)
    }()

    results := make([]Result, 0, len(jobs))
    for result := range resultChannel {
        results = append(results, result)
    }

    if err := ctx.Err(); err != nil {
        return nil, err
    }
    slices.SortFunc(results, func(a, b Result) int {
        return cmp.Compare(a.Index, b.Index)
    })
    return results, nil
}

This example uses integer ranges available in modern Go. A traditional for workerID := 0; workerID < config.Workers; workerID++ works across older language versions.

The result contract chooses all-or-error on command cancellation. Per-URL failures remain inside Result. Another application might return partial results alongside cancellation; document whichever contract you choose.

Why the result channel closes last

Several workers send results. None of them knows whether it is the final sender. The coordinator waits for all workers and then closes the channel exactly once.

The collector ranges until closure, ensuring no completed result remains unread.

Output policy

Decide what counts as failure:

  • only transport errors;
  • any status >= 400;
  • status outside a configured allow-list;
  • timeout reported separately.

Keep raw observation (Status, Err) separate from policy so callers can choose.

Practice checkpoint

Implement the pipeline, then add a --fail-status=400 flag that causes the command to exit unsuccessfully when any status is at or above the threshold. Do not cancel sibling checks for an ordinary failing status.

Summary

  • Per-request contexts bound each network operation.
  • A fixed worker count bounds simultaneous requests.
  • The coordinator owns input and result channel closure.
  • Context-aware sends prevent abandoned collectors from leaking workers.
  • Sort indexed results only after all workers complete.

Ready to wrap up?

Mark this complete when you can explain the main idea and finish the practice task.

Copyright © 2026