Supervision Trees

Supervision trees are how March programs achieve fault tolerance. When an actor crashes, its supervisor automatically restarts it according to a configurable policy.

The Idea

Most languages push you toward defensive error handling: wrap anything that might go wrong in a try/catch, check every return value, anticipate every failure mode up front. It works, but it’s a lot of code, and it’s easy to miss a case.

Supervision trees are built on a different instinct, sometimes called “let it crash” (the idea comes from Erlang/OTP, which March’s actor model draws from). It sounds backwards at first: isn’t crashing bad? The insight is that most failures are transient: a stale cache entry, a flaky connection, one bad message that corrupted a bit of local state. For that kind of bug, the cheapest reliable fix isn’t to carefully detect and repair the corruption; it’s to throw the whole thing away and start over with fresh state. So instead of handling every error everywhere, you structure your system so that:

Worker actors do their job, and simply crash on unexpected errors instead of trying to handle them
Supervisor actors watch workers and restart them with clean state when they crash
Supervisors can themselves be supervised

The result is a tree of processes where failures are isolated to the one thing that broke, and recovery is automatic: you write the “happy path” logic once, and the supervisor handles the “something went wrong” case for you, uniformly, every time.

Declaring a Supervisor

Any actor can supervise children by adding a supervise block:

actor AppSupervisor do
  state { counter : Int, logger : Int }
  init  { counter: 0, logger: 0 }

  supervise do
    strategy one_for_one
    max_restarts 5 within 30
    Counter counter
    Logger  logger
  end
end

The supervise block:

strategy: restart policy (see below)
max_restarts N within S: if more than N restarts occur in S seconds, the supervisor itself crashes (escalates to its own supervisor)
backoff base <ms> cap <ms> jitter <n>% (optional): tunes the delay between repeated restarts of the same child — see Restart backoff
Each line ActorName field_name: a child to supervise, with field_name being the state field that stores its current Pid, optionally followed by restart <type> — see Restart types

When the supervisor starts (via spawn(AppSupervisor)), it automatically spawns all listed children.

Restart types

By default every child is permanent: it comes back whether it crashed or was deliberately stopped with kill(). A trailing restart modifier changes that per child:

supervise do
  strategy one_for_one
  max_restarts 5 within 60
  Counter counter                      -- permanent (the default)
  Job     job     restart transient    -- crash restarts it; kill() retires it
  Reaper  reaper  restart temporary    -- never restarted
end

Restart type	Child crashed	`kill(child)`	Child returned normally
`permanent` (default)	restarted	restarted	not restarted
`transient`	restarted	not restarted	not restarted
`temporary`	not restarted	not restarted	not restarted

transient is the job-worker case: a worker that finishes its assignment and is stopped on purpose stays stopped, while one that dies badly is brought back.

A death that does not restart also spends none of the supervisor’s max_restarts budget, so retiring children can never escalate a healthy supervisor. Under one_for_all and rest_for_one, a temporary child caught in a batch restart is stopped with its siblings and simply not brought back; its state field keeps the dead child’s Pid, since only an actual respawn rewrites it.

Coming from Erlang/OTP? March’s permanent is not OTP’s. OTP restarts a permanent child even when it exits normally; March never restarts a child that returned normally, under any restart type. March’s permanent is therefore closest to OTP’s transient, with kill() counting as an abnormal exit, and March’s transient differs from permanent exactly in that kill() retires it. This was a deliberate choice: the normal-exit behaviour predates restart types, and changing it would silently alter every supervise block already written.

Stopping gracefully

kill(pid) is immediate: whatever was queued in the actor’s mailbox is discarded. That is the wrong tool for a deploy, which needs the opposite — stop accepting new work, let the in-flight work finish, then exit.

Actor.stop(pid, timeout_ms) does that:

let stopped = Actor.stop(worker, 5000)

The actor is marked draining: send to it returns None, so no new work is accepted.
It works off the messages already in its mailbox.
It dies a normal death — which no restart type restarts, so a stopped child does not fight its supervisor.

stop returns only once the actor has actually stopped, so a shutdown sequence can be written as straight-line code. It returns false if the actor was already dead or already stopping. timeout_ms bounds the drain: a negative value waits indefinitely, and 0 discards the queue as soon as the in-flight message returns. Actor.is_draining(pid) distinguishes “shutting down” from “dead”, which is_alive alone cannot.

Stopping a tree

Stopping a supervisor stops its children first, in reverse declaration order — the mirror of the order they were started in — each with its own shutdown budget:

supervise do
  strategy one_for_one
  max_restarts 5 within 60
  Db      db                      -- stopped last  (5s default)
  Cache   cache shutdown 1000     -- stopped second (1s)
  Api     api   shutdown infinity -- stopped first  (waits as long as it takes)
end

shutdown takes a millisecond budget, infinity, or brutal (die at once, mailbox discarded — what kill does). A child that has not finished when its budget runs out is killed. The default is 5 seconds; kill never consults this field, so it changes nothing for code that does not call stop.

Children are detached from the supervisor before being stopped, so an orderly teardown does not trigger a restart — otherwise the children would come back and the tree would never go down.

Not yet supported: a terminate-style callback. An actor cannot run cleanup code of its own at shutdown; it can only finish the messages it has. Draining is also not yet integrated with hot code reload.

Restart backoff

When the same child crashes repeatedly, the supervisor waits a little longer before each restart so a crash loop cannot burn a core. The delay doubles per consecutive crash up to a ceiling, with jitter so that siblings crashing together do not retry in lockstep, and the streak resets once a child survives a full max_restarts window. The first crash is always restarted immediately.

The curve is tunable per supervisor:

supervise do
  strategy one_for_one
  max_restarts 5 within 60
  backoff base 25 cap 5000 jitter 25%   -- these are the defaults
  Worker w
end

base <ms>: the delay before the second consecutive restart; each further consecutive crash doubles it (8 doublings maximum).
cap <ms>: the ceiling the doubling saturates at. Must be at least base.
jitter <n>%: each delay is spread by ±n% of itself. jitter 0% disables it and makes the delays exactly reproducible, which is what you want in a test.

All three are optional individually — backoff base 100 keeps the default cap and jitter — and omitting the clause entirely gives the values shown above. Set MARCH_SUP_TRACE=1 in the environment to have each restart decision print its child index, crash streak and chosen delay.

Restart Strategies

`one_for_one`

Only the crashed child is restarted. Other children continue running.

supervise do
  strategy one_for_one
  max_restarts 3 within 60
  Worker1 w1
  Worker2 w2
  Worker3 w3
end
-- If w2 crashes, only w2 is restarted

Use one_for_one when children are independent.

`one_for_all`

When any child crashes, all children are stopped and restarted.

supervise do
  strategy one_for_all
  max_restarts 2 within 30
  DbConnection db
  CacheConnection cache
  QueryEngine engine
end
-- If db crashes, db + cache + engine are all restarted

Use one_for_all when children are tightly coupled and must be in sync.

`rest_for_one`

When a child crashes, it and all children started after it are restarted. Children started before it are left as they were.

supervise do
  strategy rest_for_one
  max_restarts 5 within 60
  Config    cfg      -- started first, independent
  Database  db       -- depends on nothing
  ApiServer api      -- depends on db
  Logger    log      -- depends on api
end
-- If db crashes, db + api + log restart; cfg is left running

Use rest_for_one when later children depend on earlier ones.

A Full Supervision Example

(Originally adapted from examples/supervision_basic.march, since removed as a redundant example; this inline copy is now the canonical source and is verified directly against the compiler.)

Interpreter-only, as written. Restart itself (one_for_one etc.) is correct on both backends, but this example reaches a supervised child from outside the tree with get_actor_field(sup, …) + pid_of_int(…), and that pair crashes in a compiled binary (compiled builds also skip a child’s init at spawn(Sup)). Run it under the interpreter (march run / run_until_idle()); see Actors → App Entry Point for the same caveat in the Builtins table.

mod BasicSupervision do
  needs IO.Console

  actor Counter do
    state { count : Int }
    init  { count: 0 }

    on Inc() do
      let n = state.count + 1
      println("[Counter] count -> " ++ int_to_string(n))
      { count: n }
    end
  end

  actor Logger do
    state { entries : Int }
    init  { entries: 0 }

    on Log(msg : String) do
      let n = state.entries + 1
      println("[Logger] #" ++ int_to_string(n) ++ ": " ++ msg)
      { entries: n }
    end
  end

  actor AppSupervisor do
    state { counter : Int, logger : Int }
    init  { counter: 0, logger: 0 }

    supervise do
      strategy one_for_one
      max_restarts 5 within 30
      Counter counter
      Logger  logger
    end
  end

  fn main() do
    -- Spawn supervisor: it auto-starts Counter and Logger
    let sup = spawn(AppSupervisor)

    -- Get child PIDs from supervisor state
    let c1_int = match get_actor_field(sup, "counter") do
                   None    -> -1
                   Some(n) -> n
                 end
    let c1 = pid_of_int(c1_int)

    println("Counter alive: " ++ bool_to_string(is_alive(c1)))

    -- Use the children
    send(c1, Inc())
    send(c1, Inc())
    run_until_idle()

    -- Crash the Counter
    kill(c1)
    println("Counter alive after kill: " ++ bool_to_string(is_alive(c1)))

    -- Supervisor restarts it with a new PID
    let c2_int = match get_actor_field(sup, "counter") do
                   None    -> -1
                   Some(n) -> n
                 end
    let c2 = pid_of_int(c2_int)
    println("New counter PID: " ++ int_to_string(c2_int))
    println("New counter alive: " ++ bool_to_string(is_alive(c2)))

    -- Restarted counter has fresh state (count = 0)
    send(c2, Inc())
    run_until_idle()
  end

end

Escalation: Max Restarts Budget

If a child crashes too frequently, the supervisor gives up and crashes itself, escalating the fault to its own supervisor:

supervise do
  strategy one_for_one
  max_restarts 3 within 60  -- 3 restarts in 60 seconds → supervisor crashes
  FlakeyWorker w
end

This prevents restart storms from grinding the system to a halt. The escalation propagates up the supervision tree until either a supervisor absorbs it or the top-level supervisor crashes the whole application.

Restart Backoff

A child’s first crash restarts immediately (zero added delay), the same synchronous, zero-delay behavior as before backoff existed, so a single crash-and-recover cycle is unaffected. Only a repeat crash of the same child slot (its crash_streak exceeds 1) is delayed: the delay is 25ms << min(streak - 1, 7): 50, 100, 200, 400, 800, 1600, then capped at 3200ms pre-jitter (the shift itself saturates at 7, so 3200ms is the true upper limit, not 5000ms), with ±25% jitter on top (observed max ~4000ms) to de-synchronize a crash storm’s retries. The streak resets to 0 once the child completes a full max_restarts ... within N window without crashing again; a healed child goes back to immediate-restart behavior on its next isolated crash.

For the batch strategies (one_for_all/rest_for_one), a pending delayed restart absorbs any further crashes among covered children that arrive before it fires, instead of scheduling a second overlapping restart; the restart’s child range widens (never narrows) to cover every sibling that crashed during the pending window.

Set MARCH_SUP_TRACE=1 to print each restart decision to stderr: march: supervisor backoff child=<idx> streak=<n> delay_ms=<ms>, plus a ` (batch restart already pending, skipped)` suffix when a crash was absorbed into an already-pending batch restart instead of scheduling its own.

MARCH_SUP_TEST_STALL_MS=<ms> is a test seam for this exact race: it makes a synchronous batch restart pause (yielding, so other actors keep running) between claiming its in-flight marker and running the strategy, so a test can land a sibling’s crash inside that window by construction instead of by luck. It is read once, does nothing when unset, and exists for the runtime’s own regression suite; leave it unset in production.

Supervision Strategies Compared

Worker crashes:     W1  W2  W3
                    ↑
                  crash

one_for_one:        ↻   ok  ok    (only W1 restarts)
one_for_all:        ↻   ↻   ↻     (all restart)
rest_for_one:       ↻   ↻   ok    (W1 and later restart)

Nested Supervision Trees

Supervisors can supervise other supervisors, forming a tree:

actor TopSupervisor do
  state { web_sup : Int, db_sup : Int }
  init  { web_sup: 0, db_sup: 0 }

  supervise do
    strategy one_for_one
    max_restarts 2 within 30
    WebSupervisor web_sup
    DbSupervisor  db_sup
  end
end

actor WebSupervisor do
  state { router : Int, cache : Int }
  init  { router: 0, cache: 0 }

  supervise do
    strategy one_for_all
    max_restarts 5 within 60
    Router router
    Cache  cache
  end
end

actor DbSupervisor do
  state { pool : Int }
  init  { pool: 0 }

  supervise do
    strategy one_for_one
    max_restarts 10 within 60
    ConnectionPool pool
  end
end

A crash in the Web tier doesn’t affect the DB tier. A crash in the DB tier escalates to TopSupervisor.

App-Level Entry Point

The app declaration is a shorthand for defining the top-level supervisor of a long-running application:

mod MyService do
  actor Worker do
    state { n : Int }
    init  { n: 0 }
    on Tick() do { state with n: state.n + 1 } end
  end

  app MyService do
    Supervisor.spec(:one_for_one, [worker(Worker)])
  end
end

This is the value-level counterpart of the supervise block used everywhere else on this page: the app body evaluates to a Supervisor.Spec, where Supervisor.spec(:one_for_one, [worker(Worker), …]) states the same thing as supervise do strategy one_for_one; Worker … end inside an actor. The differences are scope and spelling: app defines the single application root (not an inner actor’s children), and the strategy is passed as the atom :one_for_one rather than the bare one_for_one keyword the block DSL uses. Use supervise to give an actor children; use app for the application’s root supervisor. See Actors → App Entry Point for the same note from the actor side.

Interpreter-only. The app / Supervisor.spec / worker / dynamic_supervisor DSL runs under march run and march test; the compiled backend rejects a call to any of them with a positioned error (it used to fail at link time with Undefined symbols: _worker). A compiled program declares its children in a supervise do … end block.

Strategies for Supervision Design

Start with one_for_one: it’s the most common and most isolated strategy.

Use one_for_all when children share state: for example, a group of actors that all read from a shared config loaded at startup. If one crashes, the shared state might be stale and all should reload.

Use rest_for_one for pipelines: if actor B depends on actor A having started first, use rest_for_one so a crash in A also restarts B.

Keep supervisors thin: a supervisor’s job is supervision, not business logic. Don’t add handlers to a supervisor actor beyond what’s needed to manage children.

Budget restarts conservatively: max_restarts 3 within 5 is aggressive; max_restarts 10 within 60 is more lenient. Match the budget to how often legitimate transient failures are expected.

Capstone: a crash-tolerant job processor

Let’s build something real by layering the pieces one at a time: each step adds exactly one capability, and you can stop at whichever level your problem needs.

Step 1: one worker

Start with a single actor that processes jobs. On a bad job it just crashes; we’ll make that survivable in the next step.

mod JobProcessorV1 do
  needs IO.Console

  actor Worker do
    state { done : Int }
    init  { done: 0 }

    on Process(job : Int) do
      -- pretend-work; a real handler might crash on a malformed job
      println("[Worker] processed job " ++ int_to_string(job))
      { done: state.done + 1 }
    end
  end

  fn main() do
    let w = spawn(Worker)
    send(w, Process(1))
    send(w, Process(2))
    run_until_idle()
  end

end

That’s the whole job processor, but if Process crashes at any point, the worker is gone and every later job is dropped.

Step 2: put it under a supervisor (crash recovery)

Wrap the worker in a one_for_one supervisor. Now a crash is recovered from: the supervisor restarts the worker (with fresh state) instead of losing it.

Interpreter-only, as written: like the Full Supervision Example above, this reads the child PID back out with get_actor_field + pid_of_int, which crash in a compiled binary. Run it under the interpreter.

mod JobProcessorV2 do
  needs IO.Console

  actor Worker do
    state { done : Int }
    init  { done: 0 }

    on Process(job : Int) do
      println("[Worker] processed job " ++ int_to_string(job))
      { done: state.done + 1 }
    end
  end

  actor JobSupervisor do
    state { worker : Int }
    init  { worker: 0 }

    supervise do
      strategy one_for_one
      max_restarts 5 within 30
      Worker worker
    end
  end

  fn main() do
    let sup = spawn(JobSupervisor)
    let w_int = match get_actor_field(sup, "worker") do
                  None    -> -1
                  Some(n) -> n
                end
    let w = pid_of_int(w_int)
    send(w, Process(1))
    run_until_idle()

    -- A crash is now survivable: kill the worker and the supervisor restarts it.
    kill(w)
    let w2_int = match get_actor_field(sup, "worker") do
                   None    -> -1
                   Some(n) -> n
                 end
    println("worker restarted, alive: "
            ++ bool_to_string(is_alive(pid_of_int(w2_int))))
    run_until_idle()
  end

end

one_for_one is the right strategy here: one worker, independent of anything else, restarted on its own. See Restart Strategies for when to escalate to one_for_all or rest_for_one.

Step 3: fan out to N workers

One worker is a bottleneck. Spawn a pool and spread jobs across it. Each worker is the same supervised actor; we just spawn several and round-robin work to them. This step and the next borrow tools from other pages; you don’t need to have read them first, just see how they slot into the same “add exactly the resilience you need” pattern.

The data-parallel shortcut for “run this over a whole list across the pool” is List.pmap: it applies a function to every element using the same actor scheduler underneath, and gives back results in the original order, as if you’d called List.map:

-- Dispatch a batch of jobs across N workers, in parallel.
fn dispatch_all(jobs : List(Int)) do
  -- Each job runs concurrently; results come back in the original order.
  List.pmap(jobs, fn job -> handle_job(job))
end

Under a supervisor you’d list several Worker children (Worker w1, Worker w2, …, each its own state field) so a crash in one doesn’t disturb the others; that’s exactly what one_for_one gives a pool.

Step 4: add backpressure so a fast producer can’t flood the pool

Backpressure just means: the slow stage sets the pace, instead of letting a fast stage pile up work faster than it can be handled. The missing piece here: if jobs arrive faster than the pool drains them, an unbounded queue grows until memory runs out. Put a Flow pipeline in front so the consumer (the pool) sets the pace; the producer only runs as far ahead as there’s capacity:

fn process_stream(jobs : List(Int)) do
  Flow.from_list(jobs)
    |> Flow.map(fn job -> handle_job(job))   -- the slow stage
    |> Flow.with_concurrency(4)              -- 4 worker actors, bounded demand
    |> Flow.collect
end

If you’d rather bound concurrency on a plain list without a pipeline, List.pmap_n(jobs, handle_job, 4) caps in-flight work the same way. Either way, the chain is now complete: one worker → supervised (persists through crashes) → a pool (throughput) → backpressured (bounded memory under load). That progression (start simple, add exactly the resilience you need) is the heart of how March systems are built.

What runs where. The plain actor/supervisor programs (Steps 1–2) execute in the interpreter via run_until_idle(). The Flow / pmap stages (Steps 3–4) produce identical results in the interpreter but only parallelise when compiled; see Parallel Collections → Interpreter vs. compiled and Flow & Backpressure.

Next Steps

Actors: the actor model basics and the concurrency decision guide.
Parallel Collections: pmap / pmap_n for fanning work across a pool.
Flow & Backpressure: bounded streaming so a fast producer can’t flood your workers.
Clustering & RPC: take a supervised app from one node to a cluster.
Linear Types: how linear types support safe actor messaging.