URL Checker — Worker Pipeline
What you will learn
By the end of this checkpoint, you will be able to:
- start exactly the configured number of workers;
- stop sends and receives when the context is cancelled;
- close result channels only after every worker returns;
- release HTTP response bodies correctly;
- reconstruct deterministic output from nondeterministic completion.
Check one URL
func check(ctx context.Context, client *http.Client, job Job, timeout time.Duration) Result {
started := time.Now()
result := Result{Index: job.Index, URL: job.URL.String()}
requestCtx, cancel := context.WithTimeout(ctx, timeout)
defer cancel()
request, err := http.NewRequestWithContext(requestCtx, http.MethodGet, result.URL, nil)
if err != nil {
result.Err = fmt.Errorf("create request: %w", err)
return result
}
response, err := client.Do(request)
result.Duration = time.Since(started)
if err != nil {
result.Err = err
return result
}
defer response.Body.Close()
result.Status = response.StatusCode
_, copyErr := io.Copy(io.Discard, io.LimitReader(response.Body, 4<<10))
if copyErr != nil {
result.Err = fmt.Errorf("read response body: %w", copyErr)
}
return result
}
Reading only 4 KiB prevents unbounded download. Because the body may contain more data, the connection may not be reusable in every case. For a checker, correctness and bounds are more important than forcing reuse of arbitrarily large bodies. A specialised transport or HEAD policy can be evaluated separately.
Redirects follow the client's policy and can change the destination. Configure CheckRedirect when redirect count or destination restrictions matter.
Worker loop
func worker(
ctx context.Context,
client *http.Client,
jobs <-chan Job,
results chan<- Result,
timeout time.Duration,
) {
for {
select {
case <-ctx.Done():
return
case job, ok := <-jobs:
if !ok {
return
}
result := check(ctx, client, job, timeout)
select {
case results <- result:
case <-ctx.Done():
return
}
}
}
}
Cancellation is observed around both job receipt and result delivery.
Coordinator owns channel lifecycle
func CheckAll(ctx context.Context, client *http.Client, jobs []Job, config Config) ([]Result, error) {
jobChannel := make(chan Job, min(len(jobs), config.Workers*2))
resultChannel := make(chan Result, config.Workers)
var workers sync.WaitGroup
workers.Add(config.Workers)
for range config.Workers {
go func() {
defer workers.Done()
worker(ctx, client, jobChannel, resultChannel, config.RequestTimeout)
}()
}
go func() {
defer close(jobChannel)
for _, job := range jobs {
select {
case jobChannel <- job:
case <-ctx.Done():
return
}
}
}()
go func() {
workers.Wait()
close(resultChannel)
}()
results := make([]Result, 0, len(jobs))
for result := range resultChannel {
results = append(results, result)
}
if err := ctx.Err(); err != nil {
return nil, err
}
slices.SortFunc(results, func(a, b Result) int {
return cmp.Compare(a.Index, b.Index)
})
return results, nil
}
This example uses integer ranges available in modern Go. A traditional for workerID := 0; workerID < config.Workers; workerID++ works across older language versions.
The result contract chooses all-or-error on command cancellation. Per-URL failures remain inside Result. Another application might return partial results alongside cancellation; document whichever contract you choose.
Why the result channel closes last
Several workers send results. None of them knows whether it is the final sender. The coordinator waits for all workers and then closes the channel exactly once.
The collector ranges until closure, ensuring no completed result remains unread.
Output policy
Decide what counts as failure:
- only transport errors;
- any status
>= 400; - status outside a configured allow-list;
- timeout reported separately.
Keep raw observation (Status, Err) separate from policy so callers can choose.
Practice checkpoint
Implement the pipeline, then add a --fail-status=400 flag that causes the command to exit unsuccessfully when any status is at or above the threshold. Do not cancel sibling checks for an ordinary failing status.
func HasFailure(results []Result, minimumFailureStatus int) bool {
for _, result := range results {
if result.Err != nil || result.Status >= minimumFailureStatus {
return true
}
}
return false
}
The worker reports facts; the CLI applies exit policy after printing every result. Cancellation remains a separate command-level error because the requested set did not finish.
Summary
- Per-request contexts bound each network operation.
- A fixed worker count bounds simultaneous requests.
- The coordinator owns input and result channel closure.
- Context-aware sends prevent abandoned collectors from leaking workers.
- Sort indexed results only after all workers complete.
Ready to wrap up?
Mark this complete when you can explain the main idea and finish the practice task.