Project — Concurrent URL Checker

Design a bounded URL checker that demonstrates goroutines, channels, cancellation, and HTTP ownership.

What you will build

A command that checks URLs concurrently while respecting explicit limits:

urlcheck --workers=5 --timeout=2s urls.txt

Example output:

200  142ms  https://example.com
404   88ms  https://example.com/missing
ERR  2001ms https://unreachable.example  context deadline exceeded

What you will learn

  • bounded worker pools;
  • channel ownership and closure;
  • request contexts and timeouts;
  • shared http.Client reuse;
  • ordered reconstruction of concurrent results;
  • cancellation and goroutine-completion tests.

Behaviour contract

  • Read one URL per non-empty line.
  • Reject malformed or unsupported schemes before starting workers.
  • Run no more than workers checks at once.
  • Apply a per-request timeout within the overall command context.
  • Produce one result for every accepted input unless the command is cancelled.
  • Print results in input order even when requests complete out of order.
  • Return a non-zero status if any URL fails according to the chosen policy.

Define types before goroutines

type Job struct {
    Index int
    URL   *url.URL
}

type Result struct {
    Index    int
    URL      string
    Status   int
    Duration time.Duration
    Err      error
}

The index preserves input ordering without forcing workers to finish in order.

Configuration

type Config struct {
    Workers        int
    RequestTimeout time.Duration
    MaxInputURLs   int
}

func (c Config) Validate() error {
    if c.Workers < 1 || c.Workers > 100 {
        return errors.New("workers must be between 1 and 100")
    }
    if c.RequestTimeout <= 0 {
        return errors.New("request timeout must be positive")
    }
    if c.MaxInputURLs < 1 {
        return errors.New("maximum input count must be positive")
    }
    return nil
}

Limits prevent accidental resource explosions. They are product decisions, not universal constants.

HTTP client ownership

Construct one configured client and reuse it across workers. http.Client and the default transport are designed for concurrent reuse.

client := &http.Client{
    Timeout: config.RequestTimeout,
}

Using both client timeout and request context can be reasonable, but understand which deadline is shorter and which error reaches the caller. In the implementation checkpoint, use one clear per-request child context and a shared client with transport settings appropriate to the exercise.

Request method choice

HEAD can avoid downloading a response body, but some servers implement it incorrectly or differently from GET. GET is more widely representative but requires strict body handling.

For this project, use GET, read only a small bounded amount if needed to enable connection reuse, then close the body. Never download unbounded content merely to check availability.

Security boundary

For a local CLI checking URLs supplied by its user, the user controls the destinations. If this capability becomes a server endpoint, arbitrary URLs create server-side request forgery risk: clients may reach loopback, cloud metadata, or private networks through your service.

Do not expose the same implementation as a public URL-check API without strict destination policy, DNS/IP validation that accounts for rebinding, redirect controls, network egress restrictions, and response limits.

Architecture

input parser
  ↓ indexed jobs
bounded jobs channel
  ↓
N workers using one HTTP client
  ↓ indexed results
result collector
  ↓ ordered output

The coordinator owns closing channels. Workers do not close shared channels.

Practice

Implement URL parsing and configuration validation. Produce a slice of indexed jobs without starting any goroutines.

Summary

  • Define observable behaviour and resource bounds before adding goroutines.
  • Index jobs so output order is independent of completion order.
  • Reuse one configured HTTP client.
  • Limit response handling and always close successful bodies.
  • Treat user-controlled server-side destinations as a serious security boundary.

Ready to wrap up?

Mark this complete when you can explain the main idea and finish the practice task.

Copyright © 2026