Control
Control: the pure core of the in-cluster control plane (distributed-deploys build step 12a). Design: specs/plans/2026-09-28-dd-step12-control-plane-design.md, sections 3, 4, 7 and 8, decisions D37-D41.
The control plane holds no root keys (D37). forge writes and signs a RELEASE: a sequence number, the digest of its parent, the environment, the topology digest, the builds it ships, and an ordered list of steps with host selectors and gates. The leader only executes it:
parse/serializeread and write the release's text form (section 4);
`check` validates it, `digest` / `verify` / `sign` cover its identity and
its operator signature, and `accept` is the leader's compare-and-set on
`parent`.
* `pass` is the executor, level-triggered and pure: its inputs are the
newest release held, the membership (which node is in which pool), what
each Agent last reported, the nodes with an order outstanding, and the
time. It finds the first step whose selected nodes do not yet report the
step's end state and orders the next nodes the step allows. Nothing else
is carried between passes, so a new leader resumes a half-finished
rollout from observation alone. A failed step or gate HALTS the release;
the executor never rolls back (a rollback is a new release).
* `Leader` wraps `pass` in the few facts a leader keeps in memory: the
reports (`Obs`, with the leader-clock time each node became healthy, so a
gate window restarts on a new leader, section 4), orders in flight, and a
sticky halt.
* `agent_apply` is the Agent's side of one `StepOrder`, over an `AgentOps`
dictionary: the node's reload server and CAS are reached through it. Until
the `reload_request` builtin exists (design section 6) the dictionary is
the seam tests and the real Agent plug into.
* The payloads of the control protocols (section 7): `AgentReport`,
`StepOrder`, `StepResult`, `CtlChunk`, and the `ControlOps` dictionary a
leader's `Ctl` session is driven by. The `Ctl` and `CtlFetch`
declarations themselves live in test/session/control_peers.march for
now. A protocol may live in a stdlib module since 2026-10-03
(specs/progress/2026-10-03-endpoints-protocol-in-nested-module.md); the
move is specs/todos/2026-09-28-dd-step12a-control-wiring.md.The leader itself is count = 1 placement over SWIM with no strict lease (D40): two leaders during a partition each drive their side toward the newest release they hold, and nodes refuse anything older (12-pre).
Types
Functions
The leader's compare-and-set (design section 9): may [r] replace [head], the newest release held? The first release is accepted whatever its parent. The held release itself is accepted again (a retry). Otherwise r.seq must be higher and r.parent must be digest(head): the parent chain is enforced by the one writer that sees every release (section 5).
RolloutApply one order on this node. Idempotent: a node that already reports the order's end state answers ok without touching the reload server, so an order a new leader sends again is harmless. Missing artifacts or a refused line answer not-ok with the reason, which halts the release. A topology step the reload server took is given a few placement ticks for the node to report it (agent_apply_within).
Apply a do:certs order on this node (step 12b): check the release it carries (the deploy key's signature, that it is the release the order names, not older than one this node took from), then take the certificate item for THIS node, if there is one (apply_cert; items for other nodes are not this node's to apply), then every revocation (ClusterNode.revoke keeps a known one once, so a revocation already gossiped here is a no-op). Idempotent: a certificate already in use and a revocation already known change nothing. Any refusal answers not-ok with the reason, which halts the release.
agent_apply_certs, remembering a failure in [memory] like agent_apply_recording, and the release's seq once it succeeds: the floor below which this node refuses certificate releases (CertOps.floor should read it back with cert_floor).
agent_apply_ticked (on a node whose placement loop ticks every [tick_ms]), remembering a failure in [memory] so the node's next report carries it (a new leader re-derives the halt from it) and forgetting it once a later order succeeds. A batch left half-staged by an earlier failed step is rolled back first.
agent_apply on a node whose placement loop ticks every [tick_ms] (MARCH_PLACEMENT_TICK_MS; agent_apply assumes its default, 200): a topology step waits topology_wait_ms(tick_ms) for the node to report it.
agent_apply, waiting at most [wait_ms] (polling every [poll_ms]) for a topology step's end state. The reload server takes a TOPOLOGY line at once, but the node reports the digest only once its placement loop has re-read the file, on its next tick (MARCH_PLACEMENT_TICK_MS): until then the step is pending, not failed. Every other step is checked once.
The non-empty lines of a reload-server answer.
The audit line for a release's end: a halt (with its step, node and reason) or completion; None for any other decision.
The audit line for an order sent to a node.
The result of a release the leader refused, from its reason: err_stale, err_fork, err_parent (the compare-and-set on parent), or err_invalid (the release does not check).
The audit line for a release offered to the leader: [result] is \
The audit line for a node's answer to an order.
The newest release seq this agent has taken certificates or revocations from (0: none), as agent_apply_certs_recording keeps it.
A node's certificate state as it reports it (AgentReport.certs): cert:<serial> for the certificate it presents (absent on a shared-secret node), then rev:<id> for every revocation it knows (NodeCert.revocation_id).
Is the release well formed? A positive seq, a parent, env and topology, distinct builds, steps numbered 1..n in order, every activate(b) naming a shipped build, a drain line for a drain step, every signed line naming a step that exists, and certificate and revocation items that decode, one certificate per node, with a do:certs step to deliver them.
Split [hex] (an artifact as hex) into chunks of [size] hex characters; an empty artifact is one empty last chunk.
The release's content address: sha256 (hex) of signed_text. A child names it as its parent.
A leader holding nothing yet.
Run a pass. The orders it returns are now in flight. A RolloutHalted verdict sticks until a new release; RolloutComplete and RolloutWait hold no state.
Offer a release to the leader (check, then accept's compare-and-set). A new release clears the halt.
An agent reported: record it; the node's order (if any) is no longer in flight. A failure this leader already knows of for the node is kept even if the report omits it, so a halt cannot be lost to a report sent before the failure; so is the node's detail when the report has none (a poll's).
An agent answered an order. A failure is recorded as the node's failed step (so the next pass halts; a new leader re-derives the halt from the agent's own report). The order stays in flight until the node's next report, so it is not sent again on the strength of a stale report.
The artifacts of [o] the node does not hold yet (to fetch over CtlFetch before applying).
The parent of the first release of an environment.
Record [rep] at leader time [now]. A node's healthy window starts when the leader first sees it healthy in its current state, and restarts when the state changes or the node reports unhealthy. The window is the leader's own, so a new leader restarts every window: conservative (design section 4).
The order that takes [step] of [r] on [node].
The artifact hashes [o] needs.
Parse a release document (design section 4). The first non-blank line must be release v1; words may be separated by any run of spaces, and sig - stands for an unsigned release. The result is checked, so an Ok release is well formed (not yet verified: see verify).
A duration: 250ms, 60s, 2m, or a bare number of milliseconds.
The executor's pass (design section 4), a pure function of the release, the membership, the observations, the nodes with an order in flight, and the leader's clock. Steps run strictly in order: a step's orders go out only once every earlier step is done on its nodes and its gate has passed, and a node is ordered only while it does not report the step's end state and has no order in flight. So no step is skipped and none is applied twice.
The Agent's ops over this node's reload server. [status] is called for each report: the topology status text (Topology.status_text), and whether the verified topology is the one the node applied. [memory] keeps the last failed step (see agent_apply_recording). [certs] is the node's certificate state (certs_text). has is the artifact's presence in the CAS (a base: in the release is informational: the base image is not an artifact).
The reload server's answer to [line] (or lines: the request is a stream, a body verb's bytes follow the first newline).
What the node reports without the status file: the reload server's NODE_STATE and PINS, and [failure] (the last step this agent failed, seq step why, or \"\"). Not VERSIONS_DETAIL: one line per function (about 14,000 in a small app), on every poll, cost a node gigabytes within a minute. The patch artifacts in effect (NODE_STATE) are what the executor needs, and an operator reads the per-function view through the control API (VERSIONS_DETAIL passes through).
The last step [rep]'s node failed, if any.
The serial of the certificate [rep]'s node presents (\
The builds [rep] runs.
Does [rep] show the end state [w] of an order with action [action] (show_action form)?
The nodes [step] applies to, given the selections of the steps before it ((step, nodes) pairs): canary(n) the first n targets, all every target, rest the targets no earlier step with the same action selected. Selection depends only on the release and the membership, never on progress, so every leader selects the same canary.
The release document: signed_text, then its sig line (sig - when unsigned). parse(serialize(r)) == Ok(r) for a well-formed r.
The action as the release and the wire write it: activate(web), topology, drain.
The decision as one line, for status output and logs.
A duration written the way parse_duration reads it back: whole seconds as Ns, else Nms.
Sign [r] with an ed25519 secret key (ed25519_seed_keypair form). What forge does; here for tests and tooling.
The canonical text the signature covers: the release without its sig line, one entry per line, single spaces, a newline after every line. forge signs exactly this, so a document may be laid out any way parse accepts and still verify.
The value of key:value among the words of [line] (\
The STATE line of a NODE_STATE answer (\
A step's target nodes: the members in any of its pools, in node order (the canary is the first).
How long a topology step waits for the node's placement loop to apply a digest the reload server already took, given the loop's tick [tick_ms]: ten ticks, at least two seconds (a loaded host delays a tick), at most a minute.
Does [r] carry a valid operator signature for [pubkey]? The leader checks this before storing a release so it never relays junk; the nodes check the signed lines themselves either way (the control plane is not trusted, D37).
The canonical text of what a node runs: build=manifest words, sorted by build.
The end state a node reports once [step] of [r] is done on it (StepOrder.want).
The end state [node] reports once [step] of [r] is done on it: want, except for a do:certs step, whose end state is per node: cert:<serial> when the release carries a certificate for [node], and rev:<id> for each of its revocations (the words of certs_text the node must report).
[rep] with its failure set to [f].