Graceful Shutdown
What you will learn
By the end of this lesson, you will be able to:
- convert process signals into cancellation;
- distinguish stopping new work from completing active work;
- coordinate HTTP, background workers, and owned resources;
- apply a bounded shutdown deadline;
- avoid common goroutine and deployment shutdown races.
Mental model: shutdown is a state transition
A service does not jump directly from running to gone. A controlled shutdown normally moves through stages:
ready
↓ mark unready
draining
↓ stop intake and wait
stopping
↓ close owned resources
stopped
The exact order depends on the service, but it must be intentional.
Receive cancellation at the process boundary
func main() {
ctx, stop := signal.NotifyContext(
context.Background(),
os.Interrupt,
syscall.SIGTERM,
)
defer stop()
if err := run(ctx); err != nil {
fmt.Fprintln(os.Stderr, err)
os.Exit(1)
}
}
SIGTERM is commonly used by process managers on Unix-like systems; signal availability and semantics vary by platform. The application context expresses the request to stop.
Serve and wait for either failure or cancellation
func serve(ctx context.Context, server *http.Server) error {
serveErrors := make(chan error, 1)
go func() {
serveErrors <- server.ListenAndServe()
}()
select {
case err := <-serveErrors:
if errors.Is(err, http.ErrServerClosed) {
return nil
}
return fmt.Errorf("serve HTTP: %w", err)
case <-ctx.Done():
shutdownCtx, cancel := context.WithTimeout(context.Background(), 15*time.Second)
defer cancel()
if err := server.Shutdown(shutdownCtx); err != nil {
return fmt.Errorf("shutdown HTTP server: %w", err)
}
return nil
}
}
Shutdown closes listeners, closes idle connections, and waits for active connections to become idle, bounded by the context. It does not wait for every possible hijacked or upgraded connection; applications using those features must coordinate them separately.
Use a fresh bounded context for cleanup. The main application context is already cancelled.
Mark unready before draining
In an orchestrated deployment:
- Mark the instance unready.
- Allow routing state to propagate for a short, configured drain period if the platform requires it.
- Stop accepting new connections or requests.
- Wait for active requests within the shutdown budget.
The total must fit inside the platform's termination grace period. Do not guess: configure and test it in the actual environment.
Stop background workers
type WorkerGroup struct {
cancel context.CancelFunc
wait sync.WaitGroup
}
func (g *WorkerGroup) Start(parent context.Context, worker func(context.Context)) {
ctx, cancel := context.WithCancel(parent)
g.cancel = cancel
g.wait.Add(1)
go func() {
defer g.wait.Done()
worker(ctx)
}()
}
func (g *WorkerGroup) Stop() {
if g.cancel != nil {
g.cancel()
}
g.wait.Wait()
}
Real groups need clear rules for multiple workers, repeated calls, startup failures, and bounded waiting. A worker must actually observe ctx.Done() around blocking operations.
Choose shutdown order from ownership
A common order is:
- mark unready;
- stop external intake;
- wait for active handlers;
- cancel internal producers;
- let consumers drain or stop according to delivery guarantees;
- flush bounded telemetry;
- close database and other owned clients.
If handlers still need the database while draining, closing it before server.Shutdown completes creates failures. Ownership and dependency direction determine the order.
Delivery semantics matter
A queue worker may need to acknowledge a job only after durable completion. Cancellation halfway through must not falsely mark it successful. Design jobs to be idempotent when redelivery is possible.
Shutdown cannot guarantee completion after a hard process kill or machine failure. Durable systems need recovery and replay semantics, not only graceful process code.
Common mistakes
- Calling
server.Closefirst: active connections are closed immediately instead of drained. - Using the cancelled request context for cleanup: cleanup ends immediately.
- Closing the database before handlers finish: in-flight requests fail during the drain.
- Waiting forever: deployments hang because one dependency or goroutine never returns.
- Using
time.Sleepas worker coordination: completion remains unknown. - Assuming graceful shutdown covers crashes: durable recovery is still required.
Practice
Build a service with one HTTP server and two background workers. On cancellation, mark readiness false, drain HTTP within ten seconds, cancel workers, wait for them within the remaining budget, then close storage. Record each transition.
func (a *Application) Shutdown() error {
a.readiness.Set(false)
ctx, cancel := context.WithTimeout(context.Background(), a.shutdownTimeout)
defer cancel()
var problems []error
if err := a.server.Shutdown(ctx); err != nil {
problems = append(problems, fmt.Errorf("HTTP shutdown: %w", err))
}
a.cancelWorkers()
if err := waitGroupContext(ctx, &a.workers); err != nil {
problems = append(problems, fmt.Errorf("worker shutdown: %w", err))
}
if err := a.database.Close(); err != nil {
problems = append(problems, fmt.Errorf("close database: %w", err))
}
return errors.Join(problems...)
}
waitGroupContext requires careful implementation because sync.WaitGroup has no context-aware wait method. One approach has a helper goroutine close a channel after Wait; select between that channel and ctx.Done(). Ensure repeated shutdown is safe, and decide whether a timed-out process should return for the process manager to enforce its hard deadline.
Summary
- Shutdown is an ordered transition, not one function call.
- Mark unready and stop intake before releasing dependencies.
- Use a new bounded context for cleanup after cancellation.
- Coordinate every goroutine and blocking operation with an explicit stop path.
- Durable recovery and idempotency are still needed for crashes and hard kills.
Ready to wrap up?
Mark this complete when you can explain the main idea and finish the practice task.