Project — Concurrent URL Checker
What you will build
A command that checks URLs concurrently while respecting explicit limits:
urlcheck --workers=5 --timeout=2s urls.txt
Example output:
200 142ms https://example.com
404 88ms https://example.com/missing
ERR 2001ms https://unreachable.example context deadline exceeded
What you will learn
- bounded worker pools;
- channel ownership and closure;
- request contexts and timeouts;
- shared
http.Clientreuse; - ordered reconstruction of concurrent results;
- cancellation and goroutine-completion tests.
Behaviour contract
- Read one URL per non-empty line.
- Reject malformed or unsupported schemes before starting workers.
- Run no more than
workerschecks at once. - Apply a per-request timeout within the overall command context.
- Produce one result for every accepted input unless the command is cancelled.
- Print results in input order even when requests complete out of order.
- Return a non-zero status if any URL fails according to the chosen policy.
Define types before goroutines
type Job struct {
Index int
URL *url.URL
}
type Result struct {
Index int
URL string
Status int
Duration time.Duration
Err error
}
The index preserves input ordering without forcing workers to finish in order.
Configuration
type Config struct {
Workers int
RequestTimeout time.Duration
MaxInputURLs int
}
func (c Config) Validate() error {
if c.Workers < 1 || c.Workers > 100 {
return errors.New("workers must be between 1 and 100")
}
if c.RequestTimeout <= 0 {
return errors.New("request timeout must be positive")
}
if c.MaxInputURLs < 1 {
return errors.New("maximum input count must be positive")
}
return nil
}
Limits prevent accidental resource explosions. They are product decisions, not universal constants.
HTTP client ownership
Construct one configured client and reuse it across workers. http.Client and the default transport are designed for concurrent reuse.
client := &http.Client{
Timeout: config.RequestTimeout,
}
Using both client timeout and request context can be reasonable, but understand which deadline is shorter and which error reaches the caller. In the implementation checkpoint, use one clear per-request child context and a shared client with transport settings appropriate to the exercise.
Request method choice
HEAD can avoid downloading a response body, but some servers implement it incorrectly or differently from GET. GET is more widely representative but requires strict body handling.
For this project, use GET, read only a small bounded amount if needed to enable connection reuse, then close the body. Never download unbounded content merely to check availability.
Security boundary
For a local CLI checking URLs supplied by its user, the user controls the destinations. If this capability becomes a server endpoint, arbitrary URLs create server-side request forgery risk: clients may reach loopback, cloud metadata, or private networks through your service.
Do not expose the same implementation as a public URL-check API without strict destination policy, DNS/IP validation that accounts for rebinding, redirect controls, network egress restrictions, and response limits.
Architecture
input parser
↓ indexed jobs
bounded jobs channel
↓
N workers using one HTTP client
↓ indexed results
result collector
↓ ordered output
The coordinator owns closing channels. Workers do not close shared channels.
Practice
Implement URL parsing and configuration validation. Produce a slice of indexed jobs without starting any goroutines.
func ParseURLs(reader io.Reader, maximum int) ([]Job, error) {
scanner := bufio.NewScanner(reader)
jobs := make([]Job, 0)
for scanner.Scan() {
raw := strings.TrimSpace(scanner.Text())
if raw == "" {
continue
}
if len(jobs) == maximum {
return nil, fmt.Errorf("input exceeds maximum of %d URLs", maximum)
}
parsed, err := url.ParseRequestURI(raw)
if err != nil || (parsed.Scheme != "http" && parsed.Scheme != "https") || parsed.Host == "" {
return nil, fmt.Errorf("line %d contains an invalid HTTP URL", len(jobs)+1)
}
jobs = append(jobs, Job{Index: len(jobs), URL: parsed})
}
if err := scanner.Err(); err != nil {
return nil, fmt.Errorf("read URLs: %w", err)
}
return jobs, nil
}
For precise source line numbers, maintain a separate line counter because skipped blank lines make len(jobs)+1 inaccurate. That improvement is part of the exercise review.
Summary
- Define observable behaviour and resource bounds before adding goroutines.
- Index jobs so output order is independent of completion order.
- Reuse one configured HTTP client.
- Limit response handling and always close successful bodies.
- Treat user-controlled server-side destinations as a serious security boundary.
Ready to wrap up?
Mark this complete when you can explain the main idea and finish the practice task.