ClusterNode

The advertised address of [node_id], if this node has heard it.

Every member this node knows of, in any status, itself included.

Attach a cluster dictionary. With start, the only way to obtain a Cap(ClusterNode.Live); this one is the test seam: a test builds ops (usually { ClusterNode.ops_stub(id) with members: ..., subscribe: ... }) and hands the cap to code written against a real node.

How a node named [name] authenticates, from the environment, for code that opens connections of its own (SessionNode's direct runner, run_<Role>): certificate mode when MARCH_NODE_CERT is set (with MARCH_NODE_KEY and MARCH_CLUSTER_OPERATOR_PUBKEY, each a value or a file, and MARCH_CLUSTER_REVOCATIONS for the revoked list), exactly as config_from_env reads them; otherwise ClusterAuth.Secret([secret]).

May a node that presented [cert] play role [role] of protocol [proto] in [mode] ("offer" or "initiate")? Always Ok on a shared-secret node; on a certificate node, SessionAP.authorize against the peer's certificate (None: the peer presented none). What SessionNode's access points check when a session forms.

Wait (polling every 50 ms) until at least [n] other nodes are Alive, or [timeout_ms] passes. True if they are.

What a CERT_UPDATE's proof signs: the sender, the receiver, the link's handshake transcript [binding], the update's [counter] on that link, and the certificate's wire bytes.

Is this node in certificate mode? Every authorization check (the two-way role check at session formation, raw-send denial, registry visibility) applies only then; a shared-secret node checks nothing, as before step 11b.

A configuration with the defaults: SWIM period 1 s, ack timeout 500 ms, suspect timeout 3 s, 2 indirect helpers, tick 100 ms, redial backoff

1. [advertise] ""

means `127.0.0.1:<listen_port>`. The SWIM timings trade detection speed
for false suspicions on a loaded host (a 300 ms ack timeout suspected
healthy peers on a busy CI runner); override `swim` / `period_ms` with a
record update to tune them. The environment can move the three SWIM
defaults without a rebuild: `MARCH_SWIM_PERIOD_MS`, `MARCH_SWIM_ACK_MS`
and `MARCH_SWIM_SUSPECT_MS` (each a positive integer; anything else keeps
the default). A record update still wins over them.

A name unregistered anywhere in the cluster leaves a tombstone in every
replica of the registry; it is dropped `tombstone_grace_ms` after it was
first seen (default 3 hours, `MARCH_REGISTRY_TOMBSTONE_GRACE_MS`). Keep it
well above the longest partition the cluster should heal from: a partition
longer than that can bring back a binding whose holder is gone, which the
node service then hides and its owner unregisters again.

Configuration from the environment: MARCH_NODE_NAME, MARCH_NODE_PORT, MARCH_NODE_ADVERTISE (default 127.0.0.1:<port>), MARCH_CLUSTER_NODES (seeds, host:port,...), MARCH_CLUSTER_SECRET, MARCH_NODE_CREATION (default: the current time in ms, which grows across restarts on one host).

Certificate mode: with MARCH_NODE_CERT set, the node authenticates with a
certificate instead of the secret, and MARCH_NODE_KEY and
MARCH_CLUSTER_OPERATOR_PUBKEY must be set too (each may name a file
holding the value). MARCH_CLUSTER_SECRET is then ignored: both set means
certificate mode. Without MARCH_NODE_CERT the node runs in shared-secret
mode, as before (MARCH_NODE_KEY alone is an error, not a silent fallback).
MARCH_CLUSTER_REVOCATIONS (certificate mode; tokens or a file of them)
seeds the node's revocation list. When MARCH_NODE_CERT names a file, the
node re-reads it (and MARCH_NODE_KEY's file) every
MARCH_NODE_CERT_POLL_MS (default 10000) and replaces its certificate live
when they change (`replace_cert`; `cert_files`).

The control-connection frame tags, all ClusterNode's own and exempt from raw-send denial: SWIM 0-3, REGISTRY_SYNC_REQ/RESP 5-6, MONITOR_REQ 7, MONITOR_FIRE 8, DELIVERY_FAILED 10, CREDIT 11, MONITOR_ACK 12, MEMBER_GOSSIP 13, MEMBERS_SYNC 14, REVOCATIONS 15, CERT_UPDATE 16. (9 is ACTOR_MSG, the only frame on a data connection, and the only one checked.)

Recheck every linked peer's certificate at [now_s] (unix seconds): drop the link of one that has expired or been revoked, and queue the member Dead with that reason (the tick that follows reports it).

A dial to [addr] failed. [refused]: nothing listens there -- for a member we know at that address that is not already Dead, that is definite (PeerDown: Dead at the next tick, cause "connection refused"). Either way the address backs off, doubling to redial_max_ms.

A pair WE dialled to [dialed] ("host:port" as dialled -- a seed may be spelled differently from the peer's advertised [peer_addr]) is up: record which node that address reaches, clear its dial state, and link it.

core_dialed, for a link whose control-connection handshake transcript is [binding] (NetKernel.link_binding).

A control frame from [from] (tag [tag]). SWIM pings are answered at once; acks, gossip and member syncs are queued for the next tick; a ping-req is relayed (we ping the target, and pass its ack back to the requester). Frames of other families are ignored here (the registry and the data plane handle theirs).

A connection of the current pair to [id], generation [gen], ended. A close from a replaced generation is ignored. Otherwise the link is gone, the member is suspected (a hint, not a verdict) and redialled at once.

A pair to [id] is up ([cfd]/[dfd], opened by [dialer], its control writer [cw]). Applies the duplicate rule, gives it a generation, records what the handshake said about the peer (name, advertised address, creation) as an Alive observation. A completed authenticated link directly proves a peer previously declared Dead or Suspect is alive, so its observation advances that peer's incarnation; otherwise a split brain can leave both sides retaining Dead records that neither can independently refute. It then sends our whole member view.

core_linked, for a link whose control-connection handshake transcript is [binding] (NetKernel.link_binding).

A locally registered process died: unregister every name it held.

The certificate peer [id] presented in its latest verified handshake (certificate mode); what peer_cert answers and step 11b checks.

A CERT_UPDATE from linked peer [from], at [now_s] (unix seconds): the certificate must verify under the operator key, be unexpired and not revoked, name the same node as the one [from] presented before, not be issued before it (NodeCert.supersedes), and come with a proof by its own key for this sender and receiver, THIS link's handshake transcript and a counter above the last update taken on this link. Ok with the new state (holding it as [from]'s certificate) and the certificate, or Err with why it was refused.

Has a peer's registry been merged here (or did this node start with no seeds to wait for)? For tests of the core.

Register [name] -> local pid [pid] (its Down-watcher is [watcher]). Refused if another holder's binding is visible here -- the API checked the mirror first, so that is a race lost: the watcher is retired and the name's watchers get Lost(name, holder).

The recorded registrant of binding [name] (the certificate identity of the node that registered it, as this replica recorded it; "" when unknown), if the registry holds a present binding for it. Unlike lookup, not hidden.

How many peer member views have shown this node as rejoining (self_stale); for tests of the core.

This node's certificate is replaced by [signed], whose key is [sk] (both already checked: replace_cert). Dials handshake with it from now on, and every linked peer is sent a CERT_UPDATE. A shared-secret node is unchanged.

Add a revocation (from the API or a peer): kept and passed on to every linked peer if it is the operator's and new here, ignored otherwise. The certificates it names are dropped at the next tick.

A dial of [addr] reached this node itself (a seed list naming us): never dial it again.

Unregister [name] if this node holds it (compare-and-delete).

This node's creation (its incarnation number, part of every global pid it issues).

Certificate credentials for node [name] from their text forms: the node's ed25519 secret key (hex, 64 bytes), its certificate (NodeCert text form) and the operator public key (hex, 32 bytes). The certificate must verify under the operator key now, name [name], and name the secret key's public key -- a misconfigured node fails here, with the reason, rather than at every handshake.

A CERT_UPDATE frame's certificate, counter and proof.

Frames from this node's peers dropped because they failed their MAC (sealed connections: see specs/lang/clustering.md, "Per-frame MAC").

A global pid for the local actor [pid] on this node (to hand to a peer as a reply address).

Run the cluster link's authenticated handshake (NetKernel.handshake_auth) on [fd], a connection that is not a cluster link, with this node's identity and credentials as of now: the peer must hold the cluster secret (shared-secret mode), or an operator-signed, unexpired, unrevoked certificate whose key it proves (certificate mode; the result's cert). [role] names the connection in the hello. Both ends call it. Used by the control API's candidate-to-candidate verbs (AUDIT_COPY, RELEASE_COPY); call NetKernel.forget(fd) when done, before the fd is closed.

Is [type_tag] session traffic (session_tags(), ignoring a \

How many peers this node currently has a connection pair to.

The process bound to [name] cluster-wide, as this node sees it (a Vault read). In certificate mode a binding is hidden unless a raw send to its holder would be allowed (both certificates carry raw_send; this node's own bindings and the coordination namespaces reference_namespaces() always show): a name you cannot raw-send to is not a reference you should hold.

Every other node this node knows as Alive.

Monitor the remote process [target]: [on_down] gets its DownReason once, when it exits (Normal/Killed/Crash) or when its node is declared dead (NodeDown). The fire is at-least-once on the wire and delivered once here. Err when this node has no link to [target]'s node.

The name of [node_id], if known.

Every name with a visible binding here.

A per-node counter: the next value (from 1) under [key]. Code that needs ids unique on this node (SessionAP's session numbers) draws them here.

This node's id.

Call f for every DELIVERY_FAILED a peer sends back, except for a session's frames (a negative seq, session_seq): those go to the session's own handler (on_session_failed).

Call f whenever a data connection ends, from that connection's reader, after its last delivery. [why] is "node <name> dead: <cause>" when SWIM declared the peer dead, "replaced" when a duplicate pair was closed, else "connection lost". Returns an id for off_peer_closed.

Call [f] on every security event: a handshake refused (the peer's certificate did not verify, it runs in the other mode, the wrong secret, ...). Returns an id for off_security_event.

Call f for every DELIVERY_FAILED a peer sends back for a frame sent with a session_seq under [key] (a frame the peer could not route: its endpoint is gone, a stale creation, or refused). Replaces any handler under [key]; off_session_failed removes it.

A dictionary for node [node_id] in which every operation panics, naming itself. Override the fields a test uses with record update.

This node's own certificate (certificate mode), or None (shared-secret mode). ops_stub's answer is None, a shared-secret node: every step-11b check is off unless a test overrides it.

The certificate peer [node_id] presented in its latest verified handshake (certificate mode), or None (shared-secret mode, or no such peer). The step-11b checks (session roles, raw sends, registry visibility) read it.

The outbound queue of the data connection to [node_id], for Node.enqueue(q, to, msg, policy) (the typed send) or NodeQueue.cast. Every write to a data socket goes through this one writer, so this is the only way to send on it: Node.send(peer, ...) writes the socket directly and would interleave with the queue.

This is a RAW send path: in certificate mode it is None (counted in
`raw_refused`, reported as `RawSendRefused`) unless this node's and the
peer's certificates both carry `raw_send`. The peer refuses a raw frame
it receives from a node without raw_send in any case.

May a raw ACTOR_MSG cross a link between a node holding [own] and a peer that presented [peer]? Always in shared-secret mode ([own] None); in certificate mode only when both certificates carry raw_send.

Raw sends this node refused (certificate mode, step 11b): outbound ones (send_msg's Err(NotAuthorized), queue_for's None) and inbound raw frames dropped from a peer. Each is also a RawSendRefused security event.

Is [name] in a namespace lookup never hides (reference_namespaces())?

The registry name prefixes exempt from certificate-mode hiding: SessionNode's and Topology's coordination names.

Register [name] -> [pid] (a process on this node) cluster-wide. Refused with Taken(holder) if another holder's binding is visible here, or another local pid's registration of [name] is still pending (so two concurrent local registrations of one name cannot both succeed). Ok means the binding is made here and pushed to every peer; it is not a lock -- a node on the far side of a partition can hold the same name, and after the heal one binding wins everywhere (watch the name to hear Lost). The check is against this node's view; if another binding reaches the node first, the registration is refused there and the name's watchers get Lost(name, holder). A process already dead by then is not registered. The process is monitored: its death unregisters the name.

Who registered [name]: the certificate identity (URI) of the registering node as this node recorded it, "" when unknown (shared-secret mode, or a relayed binding whose holder this node has not verified). None when no present binding exists. Not hidden the way lookup is.

Has this node merged a peer's registry replica yet (the sync a peer sends when a link comes up)? Always true for a node started with no seeds: it has no one to wait for. Until then this node's view of the names (and of Topology's eligibility markers) is its own alone.

When (unix ms) a peer last showed this node as rejoining: its member view held this node at an earlier creation (a restart) or Dead (a restart that kept its creation, a heal). That peer reports the node's return as NodeRejoined and lets it settle before counting it; None if no peer has.

Drop [name]'s pending claim (see reserve) if pid [p] holds it.

Replace this node's certificate while it runs (certificate mode; distributed-deploys step 12b). [cert_text] is the new certificate in its text form (forge cluster cert's .cert file); [key_hex] is its node key (hex, as in MARCH_NODE_KEY), or "" to keep the current key (a renewal made with forge cluster cert --node-key). It must verify under the operator key now, name this node and that key, and not be revoked; otherwise Err, and the certificate in use is kept. The certificate and key already in use are Ok and change nothing (no update is sent, no event reported).

New handshakes present it at once. Links already up are not redialled:
each linked peer is sent the new certificate over the link, with a proof
that this node holds its key, checks it as a handshake would, and holds it
for that link from then on -- so the link, and every session on it, stays
up past the old certificate's expiry. A peer too old to know the update
drops the link when the old certificate expires, and the redial then
handshakes under the new one. Reported to `on_security_event` as
`CertReplaced` (or `CertRefused`). `config_from_env` nodes also do this
on their own when the MARCH_NODE_CERT file changes (see `CnConfig`).

The synchronous half of register: decide, in the caller, whether local pid [p] may take [name] on node [node_id] (at [creation]). [names] is the node's mirrored view; [claims] holds the registrations sent to the node actor but not yet processed there (name -> pid). The claim is taken first (Vault.put_new, atomic), then the view is checked, and the node actor mirrors a binding before releasing its claim (release_claim), so two concurrent registrations of one name by different pids cannot both get Ok. The same pid registering again is allowed. On Ok the caller owns the claim and must send the Register that releases it.

Every revocation this node knows (from MARCH_CLUSTER_REVOCATIONS, revoke, or its peers).

Revocation tokens (forge cluster revoke's output) separated by whitespace or commas, each verified against the operator key.

Revoke a certificate cluster-wide: [token] is forge cluster revoke's output, signed by the operator key. The node drops any link whose certificate it names (the peer is reported NodeDead(_, "certificate revoked")), refuses that certificate's handshakes, and passes the revocation on to every peer. Err for a token that does not parse or is not the operator's, or on a shared-secret node.

Deliver every ACTOR_MSG addressed to local pid [pid] to [deliver], called in the data reader's task (it should only send). Its Err(reason) goes back to the sender as DELIVERY_FAILED. Returns the key for unroute.

A route is a raw receiver: in certificate mode a delivery from a peer is
refused before it reaches [deliver] (DELIVERY_FAILED to the sender, counted
in `raw_refused`, reported as `RawSendRefused`) unless this node's and the
sender's certificates both carry `raw_send`.

As route, for a SESSION endpoint (SessionNode's parties, access points and initiator inboxes): it takes session traffic (session_tags()) from any certified peer, raw_send or not, and refuses anything else from a peer where raw sends are refused. The session layer checks the rest: frames carry the session id, and access points check certificates' roles.

As route, for every message of [type_tag] with no pid route (typed receivers).

Send [payload] (an encoded message of type [type_tag]) to the remote process [to] through its node's data queue. Ok(seq) means admitted, not delivered: a delivery failure comes back as DELIVERY_FAILED (see on_delivery_failed). Err(NoConnection) when this node has no link to [to]'s node.

In certificate mode a send whose [type_tag] is not session traffic
(`session_tags()`) is a raw send: `Err(NotAuthorized)` (counted in
`raw_refused`, reported as `RawSendRefused`) unless this node's and the
peer's certificates both carry `raw_send`. Sends to this node itself are
never refused.

queue_for for SESSION traffic (SessionNode's cluster links): never refused for want of raw_send, since what it carries is checked at the receiver (session frames reach only session routes). Not for raw sends: a raw frame sent through it is refused by a receiver that would refuse it anyway.

The wire seq of the [n]th frame a session sends under [key] (> 0, never reused in this process: SessionNode draws it from a process-wide counter, so a late refusal of an ended session's frame finds no handler rather than another session's). Negative, so it never collides with a user's own seqs: a DELIVERY_FAILED for it goes to the handler on_session_failed installed under [key], not to on_delivery_failed's.

The key a session_seq was made under.

The ACTOR_MSG type tags that are session traffic, exempt from raw-send denial when they reach a session route: SessionAP's invitation, withdrawal and answer, and SessionNode's frames (cluster sessions append "#<sid>"). The list is NodeSend.session_tags(), shared with the direct connections.

The CERT_UPDATE frame node [from_id] sends [to_id] on the link whose handshake transcript is [binding], as its [counter]th update there: [signed], with a proof by its key [sk].

How many present bindings of this node's from an earlier creation the replica holds (0 once retired).

Start a node: listen on cfg.listen_port, begin SWIM, dial the seeds. Returns at once -- a node that reaches no seed runs as a cluster of one and keeps redialling them (use subscribe or await_members to wait for peers). Err only if the listen socket cannot be opened. The node is a Cap(ClusterNode.Live) minted from [io]; pass it on to everything that uses the cluster.

Stop the node: close every link (the loopback too) and the listener, stop the ticker.

Call [on_event] (in the node actor's turn: it must only send) for every membership change. A subscriber added late gets NodeUp at once for every member currently Alive. Returns an id for unsubscribe.

Unregister [name], if this node holds it. A name another node holds is left alone.

Call [on_event] (in the node actor's turn: it must only send) on every change to [name]'s visible binding: Bound(name, pid), Unbound(name), or Lost(name, winner) when a binding made on THIS node is superseded by another holder's (after a partition heals, say). A watcher added while the name is bound gets Bound at once. Returns an id for unwatch.