Projects/Build a Process Supervisor

Build a Process Supervisor

Model each service as a state machine and drive it with an injected runtime and clock, so malloc-style guesswork disappears: every restart decision, backoff interval, and shutdown deadline is a value you assert. Each lesson is one concrete spec with exact states and timings: exit 0 classified clean versus nonzero as failure, on-failure restarting on a crash but never on a clean exit, backoff doubling 1, 2, 4, 8 capped at the max, giving up to Failed after exactly three restarts in the window, a dependent blocked until its dependency is Running, reverse-order shutdown, and SIGTERM then SIGKILL after the grace timeout.

30 lessonsSmall~20 min / lessonSupervision treesRestart policiesState machines
The project

What you'll build over the next 30 lessons

Over 30 lessons you build the honest core of a process supervisor - the same service-management engine that lives inside runit, daemontools, supervisord, and systemd. The whole graded core is modelled against a fake, injectable process runtime and a virtual clock: each managed service is a state machine, and you drive it with "start this", "it became ready", "it exited with code N", and "the clock advanced". That design keeps every transition, restart decision, and backoff interval exact and fully offline - no real fork or exec, no signals, no wall-clock sleeps - so the supervisor you write behaves identically in any language.

You start with the service model and its six-state lifecycle (Stopped, Starting, Running, Stopping, Exited, Failed) and a legal-transition guard, then drive services over the fake runtime: start spawns a handle, a readiness signal marks it Running, stop sends SIGTERM, and reaping an exit classifies code 0 as a clean exit versus nonzero as a failure. On that base you add the four restart policies (never, on-failure, always, unless-stopped) and derive each restart decision, exponential backoff between restarts on the virtual clock, crash-loop protection that gives up to Failed after too many restarts in a window, a dependency graph with topological start order, cycle detection, dependents that block until their dependency is ready, reverse-order shutdown, and a reconcile loop with graceful shutdown that escalates SIGTERM to SIGKILL after a grace timeout. The capstone supervises a scripted set of services and asserts the exact state timeline.

This is a teaching-grade supervisor built around the classic design shared by runit, supervisord, and Erlang/OTP supervision trees: it manages a set of services with policies, backoff, dependencies, and clean shutdown, all against a simulated runtime and clock. It is honest about where it stops: the specs never touch the operating system, so the real fork/exec of child processes, sending real POSIX signals, and reaping real PIDs with wait live only in the runnable entry point the finalize pass adds - that layer is inherently OS and standard-library specific, and it is this project's one honest caveat.

build-a-process-supervisor / lesson-01.md
Lesson 01 · The service model and state machine

The managed service

A supervisor's whole job is to keep a set of services in the state you asked for. Before it can do anything it needs a way to describe one service - its name, the command that runs it, and where it is right now. Today you build that record and give a brand-new service its starting state.

The goal

Define a service with a name and command, and confirm a fresh one starts out Stopped.

Start here - the target
TO DO
Scenario: A new service knows its identity and starts Stopped
Givena new service created with NewService("web", "run-web")
Whenits Name, Command, and State are read
ThenName is "web", Command is "run-web", and State is Stopped
Anda second service NewService("db", "run-db") is independent - its Name is "db" and it is also Stopped
Background

A process supervisor keeps long-running programs alive: it starts them, notices when they exit, restarts them under a policy, and shuts them down cleanly. Every one of those actions happens to a single unit of work called a service - a name you refer to it by, the command that launches it, and the state it is currently in. Runit calls this a “service directory”; systemd calls it a “unit”.

Today is deliberately tiny: just the record and its starting point. A service that has never been started is Stopped - it has an identity and a command, but nothing is running yet. Every later lesson moves a service between states or acts on that command, so pinning down what a fresh service looks like is where the whole engine begins.

Make it work
// the one record the whole supervisor revolves around
type Service struct {
Name string
Command string
State State // Stopped for a fresh service
}
func NewService(name, command string) *Service {
return &Service{Name: name, Command: command}
}
CheckpointDONE
You have a service record that carries a name, a command, and a starting state of Stopped. Commit and stop here.
Scope & extensions

Where this project stops - and where to go next

The supervisor core is fully real and test-driven, and the runnable demo drives genuine child processes through restart, backoff, crash-loop give-up, and signal-based shutdown, but it ships with no config-file format and only a simulated (instant) readiness check rather than a real health probe.

Extend it next
  • Add a config file format (YAML, TOML, or JSON) so you can describe your own services instead of using the hardcoded demo list.
  • Implement a real readiness check (an HTTP or TCP probe, or a pidfile) instead of marking every service ready the instant it is spawned.
  • Teach Reconcile to respect a service's pending restart-backoff timer directly, so it is safe to call repeatedly without the entry point's ramp-up then tick workaround.
  • Add a way to cancel a crash-looping service's pending restart when shutdown begins, instead of leaving it in Exited by convention.
  • Add per-service log prefixing so each child's output is distinguishable instead of interleaved on one stream.
  • Run each child in its own process group so a SIGKILL to the supervisor itself does not orphan its children.
Recommended reading

Books & references that go deeper

  • The reference for a real service manager. Read the Restart=, RestartSec=, and Type= (readiness) options - the restart policies and readiness model this project builds in miniature.

  • A tiny, elegant supervision scheme: one supervise process per service, restart on exit, clean signalling. The design writeup shows how little a correct supervisor actually needs.

  • daemontools · Daniel J. Bernstein

    The ancestor of runit and the modern supervise-one-service model. supervise, svc, and svstat show the minimal state and signalling a supervisor is built from.

  • A userland supervisor with explicit start/stop/restart states, autorestart policies, startretries, and startsecs. Its documented state machine closely parallels the lifecycle and crash-loop protection built here.

  • The origin of supervision trees: restart strategies, intensity/period crash-loop limits, and child-order start and reverse-order shutdown - the principles behind this project's backoff, give-up, and dependency ordering.

  • Requires=, After=, and Before= define the dependency graph and start/stop ordering. The topological start order and reverse-order shutdown chapter is a direct, simplified model of this.