Observing a running node

A compiled March program can answer questions about itself while it runs: which actors exist, how deep their mailboxes are, who is crashing, how busy each scheduler is, how much memory is live, and which hot-reload epochs are still pinned. It answers on a Unix socket, read-only, and forge has four commands on top of it. This page is for the person on call: how to turn it on, what every number means, and what to do when one looks wrong.

• Turning it on
• The protocol
• The verbs
• The forge commands
• Reading diagnose findings
• From inside a program: Recon and Diagnose
• Walkthrough: a node is slow, what now
• Interpreter vs compiled, and what is not there yet

Everything here is the observe tier: it reports counts, states and kinds, never message contents, actor state, or panic text. Reading actor state is a separate, later debug tier (see the end of the page).

Turning it on

Set one environment variable when you start the program:

MARCH_OBSERVE_SOCKET=/run/myapp/observe.sock ./myapp

Any compiled program serves it: no compiler flag is needed (the server is started by the runtime’s scheduler entry, runtime/march_observe.c). With neither socket variable set, nothing is started. The interpreter does not serve a socket; it answers the same questions in-process (see Recon).

Permissions. The socket is created mode 0600: only the user the program runs as (and root) can connect. The filesystem permissions are the authentication; there is no other. It is a separate socket from the hot-reload socket, with its own threads, so an observer can never block or delay a deploy.

The path. A stale socket left at the path by an earlier run is replaced. Any other kind of file there is left alone, the server does not start, and the program logs march: observe socket path exists and is not a socket. A path longer than the platform’s Unix-socket limit (103 bytes on macOS, 107 on Linux) is refused with a log line too. In both cases the program itself keeps running. The socket is unlinked when the program exits.

Limits. At most 8 connections are served at once; the ninth gets "error":"busy". At most two windowed requests (TOP over a window) run at once, so they cannot take every connection. A request line is at most 4096 bytes.

What it costs. Nothing runs until a client connects: one accept thread waits on the socket. The counters the verbs report (dispatches, messages in and out, last run time) are always kept, socket or not. Measured on Linux arm64 with bench/actors/fanin_flood.march --opt 2 (a message-bound microbenchmark), the counters cost about +0.9% at one scheduler (+0.92%, 95% interval +0.73% to +1.16%, against the pre-counter runtime; later review fixes added about 0.3% more), and nothing measurable at eight schedulers (every 8-scheduler leg was inside the A/A noise). An open, idle socket measured +0.77% (1 scheduler) and -0.54% (8 schedulers), both inside the noise. A query walks the actor table once: about 9-11 ms per 100 000 actors for the walk, about 20 ms for a full ACTORS reply. Details: specs/progress/2026-10-02-observe-r2-counters-crash-ring.md and specs/progress/2026-10-02-observe-r1-snapshot-verbs.md.

The protocol

Connect, send one line, read one line of JSON, and the server closes the connection. \r\n line endings are accepted. Verb names are upper case and case-sensitive over the raw socket (ping is an unknown_verb; forge observe upper-cases the verb for you).

echo PING | nc -U /run/myapp/observe.sock

{"proto":"march.observe/1","node":"web-1","at_ms":1791167992809,"took_us":2,"truncated":false,"data":"pong"}

Every reply is this envelope:

Error codes:

The verbs

HELP lists every verb the node serves with its arguments and tier:

{"name":"ACTORS","tier":"observe","args":"[mbox|status|epoch|pid] [n]","help":"live actors, sorted, at most n (default 100, max 10000)"}

The example replies below are real output from test/native/observe_snapshot.march (3 supervisors of 4 children each, 5 bare actors, one of them stalled with 500 messages queued, one supervised child crashed), trimmed to fit.

ACTORS

ACTORS [mbox|status|epoch|pid] [n]. Default mbox 100; n is 1 to 10 000; the two words may come in either order.

{"total":20,"shown":3,"sort":"mbox","actors":[
  {"pid":15,"type":null,"names":["hot"],"status":"waiting","mbox":500,"user_mbox":500,"held":0,
   "mbox_limit":0,"mbox_policy":"unbounded","code_epoch":1,"cap_epoch":0,"sched":0,"pinned":false,
   "draining":false,"parent":null,"children":0,"spawned_by":null,"slices":1,"msgs_in":1,"msgs_out":0,
   "crashes":0,"child_crashes":0,"idle_ms":9592},
  ...]}

total is every live actor; shown is how many rows follow. Every actor row, here and in ACTOR and SNAPSHOT, has these fields:

slices, msgs_in and msgs_out are cumulative. For a rate, use TOP with a window.

ACTOR

ACTOR <pid>: one actor in detail.

{"pid":4,"alive":true,"cap_epoch":0,
 "actor":{"pid":4,"names":["sup_one"],"status":"waiting","children":4,"child_crashes":1, ...},
 "children":[20,1,2,3],
 "spawned":[],
 "supervisor":{"strategy":"one_for_one","max_restarts":5,"window_secs":60,
               "restarts_held":1,"restart_ages_ms":[15179]},
 "terminal":null}

A dead actor answers with the kind of death and nothing else:

{"pid":0,"alive":false,"cap_epoch":0,"actor":null,"children":[],"spawned":[],"supervisor":null,"terminal":{"kind":"Crash"}}

The panic message is never shown here or anywhere in the observe tier: panic strings can carry payloads (tokens, user data). It is kept in the node’s crash ring for the planned debug tier.

TREE

{"total":20,
 "roots":[{"pid":4,"type":null,"names":["sup_one"],"status":"waiting","mbox":0,"link":null,
           "children":[{"pid":20,"status":"waiting","mbox":0,"link":"supervised","children":[]}, ...]}, ...],
 "unsupervised":[15,16,17,18,19],
 "truncated":false}

A root is an actor with no supervisor that supervises or spawned something live. Children nest under their supervisor (link: "supervised"), or, if unsupervised, under the live actor that spawned them (link: "spawned"). Actors in neither role are listed by pid in unsupervised. A node’s mbox here is waiting work (mbox + held). The tree is cut at 64 levels or 10 000 nodes, and truncated says so.

NAMES

{"names":[{"name":"hot","pid":15},{"name":"sup_one","pid":4}]}

Sorted by name. Local registrations only (no global registry).

SCHED

SCHED [window_ms]: default 200, at most 5000. With a window the observe thread samples each scheduler’s idle time, sleeps the window, samples again, and reports utilisation over it. SCHED 0 reports lifetime figures only and does not sleep.

{"schedulers":14,"window_ms":200,
 "threads":[{"id":0,"started":true,"entered":true,"dispatches":23,"idle_polls":8683,"idle_ms":10302,
             "utilisation":0.0043,"lifetime_utilisation":0.00056}, ...],
 "utilisation":0.0049,"lifetime_utilisation":0.0004,
 "live_procs":21,"procs_spawned":22,"runq":0,"stack_failures":0,"msgs_dropped":0,
 "stacks_recycled":1,"timers":2,"ctx_released":1,"procs_freed":1,"procs_awaiting_free":0,
 "metas_freed":1,"metas_awaiting_free":0}

Utilisation is 1 - idle/wall (0.0 to 1.0); idle time is only counted on the scheduler’s idle path, so a scheduler running any green thread, actor or task, counts as busy. The top-level utilisation averages the schedulers that have started. idle_ms per thread is cumulative idle time. runq is what is queued to run right now across the node; msgs_dropped is the node’s total of messages dropped at a mailbox limit. dispatches and idle_polls are a racy read of per-thread fields and only good for orders of magnitude.

MEM

{"rss_bytes":4816896,"peak_rss_bytes":4816896,"live_objects":527,"stacks_recycled":1,"queued_messages":500,"actors":20}

EPOCHS

{"current":1,"pins":[{"epoch":1,"pins":20,"current":true,"draining":false}],"slots":[],
 "counters":{"deferred":0,"converted":0,"dropped":0,"killed":0,"stopped":0,"advances":0,
             "early":0,"forced":0,"markers_live":0,"markers_lost":0}}

pins lists every epoch something still runs, with how many units pin it and whether it is draining. slots (hot-reload builds only) lists each dispatch slot: name, implementation hash, epoch, activation time and signer. counters are the hot-reload delivery counters; see Hot Code Reload.

CRASHES

CRASHES [n]: the last n crashes, newest first (default 20, at most 256; the ring keeps 256).

{"total":1,"crashes":[{"seq":1,"kind":"crash","pid":0,"type":null,"code_epoch":1,"supervisor":4,"restart":1,"at_ms":1791167983217}]}

There is no message field, by design: the observe tier never returns panic text. The ring keeps the message for the planned debug tier, which will need a separate capability.

TOP

TOP mbox|crashes|stack|slices|msgs_in|msgs_out <n> [window_ms]: the n actors highest on one attribute (n up to 10 000). Equal values come in pid order.

• mbox ranks by waiting work (mbox + held) now. No window.
• crashes ranks by crashes + child_crashes (last hour), so a crash-looping supervisor comes first. No window.
• stack ranks by committed machine-stack bytes: the memory the runtime can attribute to one actor. A green thread’s stack grows on demand and never shrinks, so this is the deepest recursion the actor has ever done, in pages (a page is 4 KiB on Linux, 16 KiB on macOS arm64, and every actor has at least one). It is 0 under the interpreter. No window. Heap bytes per actor are not tracked (the heap is shared and reference-counted, with no per-actor accounting), so there is no heap ranking.
• slices, msgs_in, msgs_out rank by how much the counter rose over window_ms (default 1000, at most 5000): two walks, the observe thread asleep between them. An actor born during the window counts from 0. An explicit window of 0 ranks by the cumulative counter instead, with no sleep.

{"attr":"mbox","window_ms":null,"total":20,
 "top":[{"pid":15,"value":500,"type":null,"names":["hot"],"status":"waiting","mbox":500,
         "stack_bytes":16384,"crashes":0,"child_crashes":0,"children":0,"link":"none",
         "parent":null,"parent_type":null,"spawned_by":null,"supervisor":null}, ...]}

value is what was ranked; window_ms is null when no window was used. Only two windowed TOPs run at once; a third gets busy.

Each row also says how the actor is supervised, so one call answers “the biggest actors, and who restarts them”:

SNAPSHOT

SNAPSHOT [sections], sections as a comma list or words, from actors, tree, names, sched, mem, epochs, crashes (default: all). Every actor section comes from one walk, so they agree with each other.

{"mem":{"rss_bytes":2834432,"queued_messages":500,"actors":20, ...},
 "crashes":{"total":1,"crashes":[ ... ]}}

Each section is that verb’s data, with fixed arguments: actors is ACTORS mbox 100, sched is SCHED 0 (lifetime only, never sleeps), crashes is CRASHES 20. This is what forge diagnose and forge status read.

WATCH

WATCH <sections> <interval_ms> <max> turns the connection into a stream: the node sends max SNAPSHOT <sections> envelopes, one line each, interval_ms apart, then closes. sections is the SNAPSHOT comma list, or all. Each frame is an ordinary envelope plus two fields, "seq" (1-based) and "of" (max), so a client can tell a finished stream from a dropped one.

printf 'WATCH mem,sched 1000 5\n' | nc -U /run/myapp/observe.sock

The bounds are part of the contract:

Frames are produced exactly as SNAPSHOT is, on the connection’s own thread: never on a scheduler thread, so a stream costs what the same SNAPSHOTs cost and does not slow the other verbs. A bad section ends the stream with the error as its only frame. Over observe_query (inside the program) WATCH answers watch_needs_socket: there is no connection to stream on.

The debug tier: signed requests

The verbs above show counts, never data. Three more show data: STATE (an actor’s state), CRASHES_FULL (crashes with their panic messages) and SOURCE (the text of a function’s source). A node answers them only when all of these hold:

1. It was built with a deploy key: march --compile --hot-reload <Mod> --signing-pubkey <base64>, the same key forge deploy hot signs patches with. Any other build answers signing_not_configured.
2. The request is signed by that key, for this process. The signature covers the whole line except the signature itself, so changing the pid or the expiry after signing gives bad_signature. Every signed request also names boot:<hex>, the 32-hex id the node’s unsigned HELLO returns (drawn at random when the process starts). A request signed for another process or node gets wrong_boot, before its nonce is spent. So a captured line is useless after the node restarts and on any other node that holds the same key. (The observe socket serves one request per connection, so the challenge is per process, not per connection as in the remote shell’s HELLO; within one process the nonce ring stops a replay.)
3. It is fresh. Each request carries a random nonce (16-64 hex digits) and a not_after_ms wall-clock expiry at most 60 s ahead. A node refuses an expired request (expired), one that expires too far ahead (not_after_too_far), and a nonce it has already seen (replay). It remembers the last 256 unexpired nonces. If all 256 are still unexpired it refuses new requests (nonce_ring_full) rather than forget one.
4. The node’s policy allows the verb. $MARCH_DEBUG_POLICY names a file that lists the allowed verbs one per line (# starts a comment). With no file, nothing is allowed (policy). The file is read on every request, so editing it takes effect at once.

The checks run in that order (signing_not_configured, bad_args, bad_signature, wrong_boot, freshness, policy): an unsigned client learns nothing about the policy. Every attempt, allowed or not, appends a line to the audit log the reload server writes ($MARCH_AUDIT_LOG, default ~/.local/share/march/audit.jsonl):

{"ts":1791223456789,"type":"debug","verb":"STATE","pid":42,"nonce":"9f3c…","signer":"<pubkey hex>","result":"ok"}
{"ts":1791223457001,"type":"debug","verb":"TRACE_START","pid":null,"nonce":"c01d…","signer":"<pubkey hex>","result":"ok","detail":"session=3 count=10 fns=2"}

detail appears on the trace verbs (the session, the limit, how many functions). An in-process caller (Trace from March code) is audited with "signer":"in-process".

forge observe builds and signs these requests for you:

forge observe --socket /run/myapp/observe.sock --state 42          # STATE, 1 s to answer
forge observe --env prod --state 42 --timeout-ms 5000
forge observe --env prod --crashes-full --count 50

forge fetches the node’s HELLO first and signs its boot id into the request, once per target.

It signs with ~/.march/ed25519_secret.key and makes each request valid for 30 s, so a node whose clock differs by more than 30 s refuses it. forge then says which way the clock is off.

A STATE reply’s data is {"pid":42,"state":"{ n: 5, tags: [x] }","error":null}. state is exactly what Actor.inspect_state returns inside the program: the state fields in declaration order, each printed by its Show, <opaque> for a field that holds functions. When the actor cannot answer, state is null and error says why:

• dead: the pid is not running.
• timeout: the actor did not answer within timeout_ms (the node allows at most 10000; forge observe --timeout-ms at most 8000, since forge waits 10 s for any reply). An actor is asked between messages, so one stuck in a long handler or a nested receive cannot answer.
• render failed: …: a field’s Show panicked. The actor keeps running.
• the actor's code changed by a hot reload …: the renderer predates a reload that may have changed the state’s layout.

The request skips the actor’s mailbox limit, so a full mailbox still answers.

Tracing boundary calls

TRACE_START, TRACE_DRAIN and TRACE_STOP (and, from inside the program, the Trace module) watch which boundary functions run, and for which actor. Boundary functions are the ones forge deploy hot can replace: your app’s, in a --hot-reload build (stdlib functions are not on the boundary and cannot be traced). Arguments are never recorded. Only {pid, function, epoch, time}, because arguments can hold anything the program holds.

A limit is mandatory. There is no spelling of TRACE_START without one, and Trace.start takes it as an argument:

Anything else is refused: no limit gives limit_required; zero, a negative number or a value over the cap gives limit_out_of_range. Nothing is clamped silently.

More bounds, whatever the limit:

• 4 sessions at most (too_many_sessions), 32 functions per session.
• A session’s ring holds n events for a MaxEvents, and ring:<r> (default 1024, at most 16384) for a Rate. A full ring drops the event and counts it (dropped); the traced program never waits for the tracer.
• A drain returns at most 4096 events (max:, default 1024).
• A session does not belong to your connection. The observe socket answers one request per connection, so there is no connection to tie it to. It lives until TRACE_STOP, until a MaxEvents limit is reached, 10 minutes after it started, or 60 seconds without a TRACE_DRAIN. Recording stops at the first of those; the node frees the session when it needs the slot (or at TRACE_STOP). A node restart ends every session.
• While no session watches a function, calling it costs one relaxed load of a word on a cache line it already reads, and a never-taken branch.

forge observe --env prod --trace-start Billing.charge,Billing.refund --max-events 200
forge observe --env prod --trace-drain 3
forge observe --env prod --trace-stop 3

Replies (the data of the envelope):

TRACE_START -> {"session":3,"limit":{"kind":"count","n":200},"ring":200,
                "functions":["Billing.charge","Billing.refund"],"lifetime_s":600,"idle_s":60}
              (a rate limit: "limit":{"kind":"rate","n":50,"per_s":1})
TRACE_DRAIN -> {"session":3,
                "events":[{"pid":42,"fn":"Billing.charge","name_id":17,"epoch":2,"at_ns":183000}],
                "recorded":1,"dropped":0,"rate_dropped":0,"pending":0,
                "done":false,"why":null,"now_ns":921000}
TRACE_STOP  -> the same without "events": the final counters

• pid is the calling actor’s pid index (what Recon and STATE use), or -1 when the caller is not an actor (main, a task).
• epoch is the code epoch of the version that ran.
• at_ns and now_ns are nanoseconds since the session started.
• recorded is the total accepted; pending is how many are not drained yet.
• done is true once recording has stopped, and why says so: "count", "expired" (10 minutes), "idle" (60 s without a drain) or "stopped". You can still drain what is left of a finished session until it is freed.
• Errors: limit_required, limit_out_of_range, unknown_function (not a boundary function of this build), too_many_sessions, not_found (no such session), plus the signed-request errors above.

From March code, with the debug capability (Actor.debug(io), or debug in the remote shell):

match Trace.start(debug, ["Billing.charge"], Trace.MaxEvents(10)) do
  Ok(s) ->
    -- ... let the program run ...
    match Trace.drain(debug, s) do
      Ok(b) -> println(Trace.format(b.events))
      Err(_) -> ()
    end
    let _ = Trace.stop(debug, s)
  Err(e) -> println("cannot trace")
end

The in-process calls need no signature (holding Cap(Actor.Debug) stands in for it, as for Actor.inspect_state), but $MARCH_DEBUG_POLICY must still list TRACE_START, TRACE_DRAIN and TRACE_STOP: they answer Unavailable(policy) otherwise. They share the same sessions and limits. Under the interpreter, and in a build without --hot-reload, they answer Unavailable("tracing_unavailable") or UnknownFunction.

Which code is running: WHICH and SOURCE

After a hot deploy, “which version of Billing.price is this node running?” has two answers, an identity and a body.

WHICH <name> (observe tier) names the identity. name is a reload slot as EPOCHS lists it (Billing.price, or an actor’s Counter_dispatch):

{"name":"Billing.price","id":2,"impl_hash":"5e9d…","baseline_hash":"41c7…",
 "patched":true,"epoch":3,"activated_at_ms":1791743066781}

impl_hash is the hash forge deploy hot diffs and VERSIONS prints. baseline_hash is the one the program was built with, and patched says whether they differ. activated_at_ms is 0 for the baseline. A name that is not a slot (a typo, a function the build does not make reloadable, a build without --hot-reload) answers not_found.

SOURCE <sig> nonce:… boot:… not_after_ms:… name:<fn> (debug tier) returns the text of the version WHICH names: {"name":…,"impl_hash":…,"epoch":…,"source":"fn price(qty : Int) : Int do\n qty * 12 + 1\nend"}. It is signed, fresh and policy-gated exactly like STATE (see the next section), and forge observe does not send it yet. $MARCH_DEBUG_POLICY must list SOURCE. Source is debug tier because it is the program’s own text: comments, names, and whatever a developer left in a function.

The text is the function’s declaration, from its first token to where the next declaration starts (so a multi-clause function is whole and comments written before the next declaration come with this one). It is the text that was compiled into that exact impl_hash; nothing is read from disk when you ask. That is also the safety: the name has to be a registered slot, SOURCE looks the text up by that slot’s current impl_hash alone, and a name containing a path (../x, /etc/passwd) is bad_args before any lookup.

Where the text comes from:

1. A --hot-reload build, node or --compile-so patch, stores each slot function’s text under its impl_hash in .march/cas/sources/ and ~/.march/cas/sources/ (each up to 256 KiB; a longer function has no source).
2. forge deploy hot uploads the text of every version it activates, and of the version it replaces when this machine has it, to the node (CAS_PUT <impl_hash> <size> kind:source, stored under the node’s ~/.march/cas/sources/), only when $MARCH_DEBUG_POLICY on the machine running forge lists SOURCE. With no such policy nothing is uploaded and SOURCE answers no_source for that version.
3. A node built on the machine it runs on already has its baseline’s text. A node built elsewhere has the baseline’s text after the first deploy that replaces it, or never (no_source).

Sources are not garbage-collected, and a hash with the same text is never rewritten. A cluster release through the control API does not carry source yet.

From inside the program (and in forge shell, where intro and debug are pre-bound), Recon.which(intro, "Billing.price") returns a FnVersion, and Recon.source(debug, "Billing.price") the text or an error code. In-process the caller’s Cap(Actor.Debug) stands in for the signature, as it does for Actor.inspect_state, but $MARCH_DEBUG_POLICY must still list SOURCE (Err("policy") otherwise), and every attempt is audited with the signer in-process.

The forge commands

All four take a target the same way:

• --socket PATH: a local observe socket (the program’s MARCH_OBSERVE_SOCKET).
• otherwise the [hot-reload] hosts in forge.toml, reached over ssh at <reload socket>.observe; --env NAME picks the [[hot-reload.env]] entries with that name. forge status also reads topology.toml.

forge observe

forge observe [REQUEST...] [--section S]... [--json] [--socket PATH] [--env NAME]
forge observe [--tui] [--panel NAME] [--interval MS] [--sort ATTR] [--pid PID] [--socket PATH] [--env NAME]
forge observe --dump FILE [--panel NAME] [--json]
forge observe --state PID [--timeout-ms MS] | --crashes-full [-n N]   [--socket PATH] [--env NAME]
forge observe --trace-start FNS (--max-events N | --rate N/SECS) | --trace-drain ID [-n MAX] | --trace-stop ID

With a request, sends it and prints the reply envelope (indented, or one line per host with --json). The request is the positional words; the first is upper-cased, so forge observe actors mbox 20 works. With no words it sends SNAPSHOT; --section (repeatable) narrows it.

forge observe --socket /run/myapp/observe.sock                 # a terminal: the interactive view; a pipe: SNAPSHOT
forge observe --socket /run/myapp/observe.sock --json          # SNAPSHOT, every section, one line
forge observe --socket /run/myapp/observe.sock actors mbox 10  # the 10 deepest mailboxes
forge observe --socket /run/myapp/observe.sock ACTOR 42
forge observe --env prod --section mem --section crashes       # every prod host, over ssh
forge observe --env prod TOP msgs_in 10 2000 --json

Exit 0 when every target answered; 1 if any failed, including an error reply (error: observe: unknown_verb).

With no request, on a terminal (or with --tui), forge observe opens the interactive view instead of printing JSON. Scripts that pipe its output are unaffected: without a terminal, or with --json, a request, or --section, it behaves as above.

The interactive view

One screen on one node, fed by a single WATCH stream (every --interval ms, default 1000, at least 250) plus, for the panel on screen, one short request on a second connection. For a remote host both go through one ssh tunnel, opened when the host is selected and closed when you move on; ssh’s own messages are discarded so they cannot draw over the screen. When the stream ends (3600 frames) or the node restarts, the view reconnects.

Keys: q or Esc quit; 1-9 or Tab switch panel; Up/Down, j/k, PgUp/PgDn, Home/End move the selection; Enter opens the selected actor; s re-ranks Actors; n cycles the hosts of --env (several [[hot-reload.env]] entries); r refreshes the panel’s own request. The Trace panel arrives with tracing (R8).

--panel NAME prints one panel as text and exits (system, load, actors, actor, supervision, names, epochs, cluster, crashes), for logs and scripts:

$ forge observe --socket /run/myapp/observe.sock --panel supervision
6 actors; trees below, supervisors first
1 Boss [boss]   waiting  mbox 0  1 kids  crashes(1h) 3
  +- 4 Fragile   waiting  mbox 0
...

Crash dumps

A node started with MARCH_CRASH_DUMP=/var/crash/web-1.json writes one JSON document there when it dies of an unsupervised panic or of one of the runtime’s own fatal errors (reference-count underflow, an internal invariant, out of memory in the runtime). The file is written to <path>.tmp and renamed, so it is either absent or complete, with mode 0600:

{"proto":"march.observe.dump/1","reason":"panic","at_ms":1791743698380,"pid":19089,
 "snapshot": { ...the SNAPSHOT envelope, every section... },
 "crashes_total":4,
 "crashes_full":[{"seq":4,"kind":"panic","pid":null,"message":"cannot parse config", ...}, ...]}

snapshot is exactly what SNAPSHOT returned at that moment. crashes_full is the crash ring with the panic text that no live observe verb carries: the dump is operator-owned, not served on a socket, and the panic text can hold payloads, so treat the file like a log. reason is panic, or the name of the runtime function that aborted.

Read it with:

forge observe --dump web-1.json                  # the panels, as text (a terminal: the interactive view)
forge observe --dump web-1.json --panel crashes  # panic text included
forge observe --dump web-1.json --json
forge diagnose --from-dump web-1.json            # findings; exit 0/1/2 as for a live node

Panels that need a question asked of a live node (Actor, Cluster) are greyed. forge diagnose --from-dump has one snapshot, so the checks that compare two (mailbox.growth, rc.climb) find nothing; crash.loop, mailbox.over_limit, sched.* and epoch.* still do. (forge diagnose --dump FILE, which writes a before/after pair, is unchanged; reading a crash dump is --from-dump.)

What does not produce a dump, and why. The dump builds JSON, which allocates, and takes the actor-table lock; neither is safe from a signal handler that may have interrupted the allocator or that lock. So no signal handler ever writes it:

• a SIGSEGV, SIGBUS, SIGILL, SIGFPE (a miscompile, a stack overflow past the guard page), SIGKILL, the kernel’s out-of-memory kill, or abort() from the C library or a linked library (glibc’s heap checks, a sanitizer report) leave no dump;
• a fatal runtime error that happens inside a reclamation critical section or on a signal stack skips it too (the data it would walk may be mid-update);
• a panic inside a supervised actor is not fatal and does not dump (the supervisor restarts it; it is in the crash ring of the next dump).

Even where a dump is attempted, the dying thread does not write it: it hands the work to a helper thread created at startup and waits for it at most MARCH_CRASH_DUMP_TIMEOUT_MS (default 3000), then dies regardless. A helper stuck behind a lock the dying thread holds costs the dump, never the exit; a timed-out dump leaves at worst <path>.tmp, never a truncated <path>. A path that cannot be written (no such directory) also costs only the dump. An unwritten dump is not an error anywhere: if there is no file, the node died some other way.

forge status

forge status [--json] [--socket PATH] [--env NAME]

One line per node. With a topology.toml, it first prints forge topology status’s report (alive, versions, drift), then the observe line for each node that has a socket; otherwise it reads the [hot-reload] hosts.

web-1: actors 1, queued 1550, rss 5 MB, busy 0%, crashes (1h) 0, deepest mailbox sink (pid 0) 1550

busy is the lifetime utilisation over all schedulers (since start, not now: use forge diagnose or SCHED for now). crashes (1h) counts the crashes in the last 20 ring entries that fall in the last hour. The deepest mailbox is named by its first registered name, else its type, else ?. A node that does not answer prints observe unavailable (<reason>). --json prints one march.status/1 document:

{"proto":"march.status/1","topology":null,"nodes":[{"name":"web-1","observe":{"actors":20,"queued":500,"rss_mb":2,"utilisation":0.0003,"crashes_hour":1,"deepest":{"pid":15,"name":"hot","waiting":500}}}]}

forge top

forge top [--sort ATTR] [-n N] [--window MS] [--once] [--json] [--socket PATH] [--env NAME]

Redraws the biggest or busiest actors in place until interrupted. --sort is mbox (default), stack (committed stack bytes), crashes, slices, msgs_in or msgs_out; the last three rank the change over --window (default 1000 ms), which is also their refresh. The others refresh every max(1 s, 4x the time the node took to answer). -n is the row count (default 20; note it is -n, not --n). --once prints one frame with no screen control, for scripts and tickets. --json prints the node’s TOP reply (every row with its supervision fields) as one line and exits. Over --env, it watches the first matching host, through the same ssh tunnel as the other observe verbs; it needs only the observe tier.

The quick answer to “which actors are biggest, and who supervises them”: forge top --once --sort stack -n 10, or --sort mbox for the deepest queues.

web-1  actors 20, queued 500, rss 5 MB, busy 0%, crashes (1h) 1, deepest mailbox hot (pid 16) 500
sorted by mbox

     PID  NAME                 TYPE               STATUS        MBOX     STACK       MBOX CRASHES  LINK       SUPERVISOR             POLICY (held/max in window)
      16  hot                                     waiting        500     16384        500       0  none       -                      -
       1                                          waiting          0     16384          0       0  supervised 4                      -
       4  sup_one                                 waiting          0     16384          0       1  none       -                      one_for_one 1/5 in 60s, 4 kids

The first line is the forge status summary. MBOX is waiting work (mbox + held), STACK the committed stack, and the column named for the sort is the ranked value. CRASHES is the actor’s own plus its children’s in the last hour. LINK is supervised, spawned or none; SUPERVISOR is the supervisor’s pid and type (or spawned by <pid>); POLICY is shown for a supervisor only: strategy, restarts currently held out of the allowed maximum within the window, and its child count.

forge diagnose

forge diagnose [--window MS] [--dump FILE] [--json] [--socket PATH] [--env NAME]

Takes two SNAPSHOTs --window apart (default 1000 ms), runs a fixed list of checks over the pair, and prints a march.diagnose/1 envelope per target:

{
  "proto": "march.diagnose/1",
  "node": "web-1",
  "window_ms": 1000,
  "findings": [
    {
      "id": "mailbox.growth",
      "severity": "critical",
      "rows": [ { "pid": 0, "mbox": 2066, "delta": 499, "names": [ "sink" ] } ],
      "next": "forge observe ACTORS mbox 20, then ACTOR <pid> for the deepest"
    }
  ],
  "coverage": {
    "ran": [ "mailbox.growth", "mailbox.over_limit", "sched.saturated", "sched.idle_imbalance",
             "crash.loop", "rc.climb", "epoch.stuck", "epoch.old_units" ],
    "unavailable": {
      "cluster.suspect": "no cluster section yet (observe plan R1.3)",
      "names.lost": "global-registry Lost events are not recorded"
    },
    "partial": {
      "mailbox.over_limit": "per-actor dropped counts are not tracked; the node total is used"
    }
  }
}

Each finding has an id, a severity (warning or critical), the rows that triggered it, and next: the command to run next.

With several targets the exit code is the worst of them. --dump FILE also writes the raw before/after pair ({"before": ..., "after": ...}) for the first target, in the same format as the test fixtures in forge/test/fixtures/diagnose/: attach it to the incident, or turn it into a regression fixture.

forge diagnose is a good health probe for a cron job or a deploy gate:

forge diagnose --env prod --json > diagnose.json; rc=$?
[ "$rc" -ge 2 ] && page-someone < diagnose.json

On the host: the release launcher

A bundle built with forge release build (see tooling) carries a bin/<app> launcher that sets MARCH_HOT_RELOAD_SOCKET (hot-reload builds) or MARCH_OBSERVE_SOCKET to run/<app>.sock[.observe] and speaks the protocol above itself: bin/<app> ping sends PING, version and status send EPOCHS and MEM. It needs nc -U, socat or python3 on the host for that. bin/<app> observe and diagnose run forge observe --socket / forge diagnose --socket when forge is installed there.

Reading diagnose findings

Each check compares the before and after snapshots. Thresholds below are exact, from forge/lib/diagnose.ml; stdlib/diagnose.march implements the same checks with the same thresholds, and both are tested against the same fixtures. “Waiting work” is mbox + held.

mailbox.growth

Fires when an actor’s waiting work rose across the window (any increase) and is at least 10 in the after snapshot. Critical if one of those actors holds more than half of the node’s queued messages and the node has 100 or more queued; warning otherwise. The rows are the five that grew most, with their depth and growth (delta).

Usually means a consumer slower than its producers: a handler blocked on something slow (a call, I/O, a sleep), a hot actor everyone sends to, or a consumer that crashed and restarted and is catching up. A large held with mbox near 0 means the actor is waiting in an Actor.call while work piles up behind it.

Next: forge observe ACTORS mbox 20, then ACTOR <pid> for the deepest. Look at its status (is it running, or waiting?), idle_ms (has it run lately?), and held. forge top --sort msgs_in tells you whether it is still draining at all. If the queue is bounded by design, give the actor a mailbox limit and a policy rather than letting it grow.

mailbox.over_limit

Fires when an actor with a mailbox limit and a drop_new or drop_old policy has waiting work at or over its limit, and the node’s dropped-message total (SCHED’s msgs_dropped) rose during the window. Warning. Actors with the block policy never trip it (their senders wait instead).

Usually means the node is shedding load: messages to that actor are being discarded. This is the policy working, but someone is losing messages.

Next: decide whether dropping is acceptable. If not, raise the limit, add consumers, or slow the producers upstream. This check is partial: per-actor drop counts are not tracked, so a drop elsewhere on the node in the same window also satisfies the second condition.

sched.saturated

Fires when some scheduler was more than 95% busy over the window (idle time under 5% of the wall time between the snapshots) and the node’s run queue was non-empty in the after snapshot. Warning.

Usually means the node is CPU-bound: there is runnable work and no idle scheduler to take it. Latency rises for everything on that node.

Next: forge observe TOP slices 10 1000 shows who is being dispatched most (or forge top --sort slices). A single actor at the top is a hot spot to split or shard; a broad spread means the node needs more cores or less work.

sched.idle_imbalance

Fires when the node has at least two schedulers with data, one more than 80% busy over the window and another less than 20% busy. Warning.

Usually means one scheduler carries the load. Common causes: work pinned to scheduler 0 (the main thread: pinned: true in the rows), or one hot actor, which can only run on one scheduler at a time. On a lightly loaded node with one busy actor this fires and is harmless.

Next: forge observe ACTORS status 20 and look at sched and pinned on the running actors; TOP slices 10 1000 for the hot one.

crash.loop

Fires when the crash ring holds 3 or more crashes with the same supervisor. Critical if 3 or more fall in the last minute (before the after snapshot’s time); warning if 3 or more fall in the last hour. Rows are the supervisors and their counts. Only supervised crashes count (an unsupervised panic has no supervisor), and only the 20 most recent ring entries are read (the snapshot’s crashes section).

Usually means a child that cannot start or keeps hitting the same bad input. When the supervisor exhausts max_restarts within its window, it gives up and the failure moves up the tree.

Next: forge observe CRASHES 20 for the kinds, pids and restart numbers, then ACTOR <supervisor> for its strategy, max_restarts, and how many restarts it is currently holding (restarts_held, restart_ages_ms). The panic text is not available from the observe tier; check the program’s own logs.

rc.climb

Fires when live heap objects (MEM’s live_objects) were at least 1000 before and grew by more than 10% across the window, while the actor count stayed within 5%. Warning. Never fires under the interpreter (no gauge).

Usually means memory is being held with no new actors to explain it: a leak, a cache without a bound, or messages piling up. Queued messages are heap objects, so this often fires together with mailbox.growth; fix that first and see whether this one goes away.

Next: take forge observe --section mem a few times over minutes. A steady climb with flat queued_messages and flat actors is a leak worth a bug report. A one-window spike that settles is usually a burst of work.

epoch.stuck

Fires when any pinned hot-reload epoch is marked draining. Warning.

Usually means a deploy is still waiting for actors to leave old code: a long-running handler, or an actor that never receives the migrate marker.

Next: forge hot-reload status; then forge observe ACTORS epoch 20 lists the actors on the oldest epoch first.

epoch.old_units

Fires when some epoch two or more behind the current one still has pins. Warning.

Usually means units still run code two or more deploys old, which every later deploy’s drain has to wait for.

Next: forge observe ACTORS epoch 20 to find them, and ACTOR <pid> for what they are doing.

Coverage

The envelope’s coverage says what was and was not checked, so an empty findings list is never read as “healthy” for things nobody looked at:

• ran: every check above.
• unavailable: cluster.suspect (the snapshot has no cluster section yet, so unreachable or suspect peers are not reported) and names.lost (global registry Lost events are not recorded).
• partial: mailbox.over_limit, which uses the node’s total dropped count because per-actor drop counts are not tracked.

Two more limits come from the snapshot itself: its actors section is the 100 deepest mailboxes (so mailbox.growth sees at most those), and its crashes section is the last 20 crashes (so crash.loop sees at most those).

From inside a program: Recon and Diagnose

The stdlib modules Recon and Diagnose ask the same questions from March code, in-process, on both the interpreter and compiled code. Every function takes a Cap(Actor.Introspect), which main mints from its IO capability with Actor.introspect(io); a module that names the capability declares needs Actor.Introspect. Nothing here returns a panic message.

ActorInfo carries pid, type_name, names, status, mbox, held, parent, children, slices, msgs_in, msgs_out, crashes and child_crashes, with the meanings in the row table. A Diagnose.Finding is { id, severity, rows, next }, where rows is a list of pids (or scheduler ids, supervisor pids, epochs) rather than forge’s objects.

The windowed functions (proc_window, scheduler_usage, Diagnose.run) take two samples and sleep the calling green thread between them, so call them from a task or an actor that can afford to wait, not from a hot handler.

A node report from main:

mod Main do
  needs IO
  needs IO.Console
  needs Actor.Introspect

  actor Worker do
    state { n : Int }
    init  { n: 0 }
    on Work(k : Int) do
      { n: state.n + k }
    end
  end

  fn report(c : Cap(Actor.Introspect)) do
    let ns = Recon.node_stats(c)
    println("actors " ++ int_to_string(ns.actors) ++ ", queued " ++ int_to_string(ns.queued))
    match ns.rss_bytes do
      Some(b) -> println("rss " ++ int_to_string(b / 1048576) ++ " MB")
      None -> println("rss unknown (interpreter)")
    end
    -- The three deepest mailboxes, as (pid, waiting).
    List.each(Recon.proc_count(c, "mbox", 3), fn p ->
      match p do
        (pid, depth) ->
          let label = match Recon.info(c, pid) do
            Some(a) -> String.join(a.names, ",")
            None -> "(gone)"
          end
          println("  pid " ++ int_to_string(pid) ++ " " ++ label ++ ": " ++ int_to_string(depth))
      end)
  end

  fn main(io : Cap(IO)) do
    let c = Actor.introspect(io)
    let w = spawn(Worker)
    let _r = Actor.register(w, "worker")
    let _ = send(w, Work(1))
    report(c)
  end
end

Rates, scheduler load and recent crashes:

mod Main do
  needs IO
  needs IO.Console
  needs Actor.Introspect

  fn busiest(c : Cap(Actor.Introspect)) do
    -- Who handled the most messages over the next half second.
    List.each(Recon.proc_window(c, "msgs_in", 5, 500), fn p ->
      match p do
        (pid, n) -> println(int_to_string(pid) ++ ": " ++ int_to_string(n) ++ " in 500 ms")
      end)
    -- How busy each scheduler was over the next second.
    List.each(Recon.scheduler_usage(c, 1000), fn s ->
      match s do
        (id, u) -> println("scheduler " ++ int_to_string(id) ++ ": " ++ float_to_string(u))
      end)
    -- Recent crashes: how, where, and which restart, never the message.
    List.each(Recon.crashes(c, 5), fn cr ->
      let sup = match cr.supervisor do
        Some(s) -> int_to_string(s)
        None -> "none"
      end
      println(cr.kind ++ " under " ++ sup ++ ", restart " ++ int_to_string(cr.restart)))
  end

  fn main(io : Cap(IO)) do
    busiest(Actor.introspect(io))
  end
end

A self-check, for a health endpoint or a periodic task:

mod Main do
  needs IO
  needs IO.Console
  needs Actor.Introspect

  fn check(c : Cap(Actor.Introspect)) : Bool do
    let fs = Diagnose.run(c, 1000)
    List.each(fs, fn f ->
      println(f.severity ++ " " ++ f.id ++ ": " ++ f.next))
    List.any(fs, fn f -> f.severity == "critical")
  end

  fn main(io : Cap(IO)) do
    let c = Actor.introspect(io)
    if check(c) do
      println("critical finding")
    else
      println("healthy")
    end
  end
end

Walkthrough: a node is slow, what now

Requests to web-1 are timing out. Start wide and narrow down.

1. Is it alive, and what does it look like?

forge status --env prod

web-1: actors 1, queued 1550, rss 5 MB, busy 0%, crashes (1h) 0, deepest mailbox sink (pid 0) 1550

It answers, nothing is crashing, and the schedulers are mostly idle over its lifetime. But 1550 messages are queued, all of them on sink. A mostly idle node with a deep queue is a consumer that is stuck, not a node that is overloaded.

2. What is wrong right now?

forge diagnose --env prod --dump /tmp/web-1-incident.json
echo $?    # 2: critical

Its findings, in short: mailbox.growth (critical; row {"pid": 0, "mbox": 2066, "delta": 499, "names": ["sink"]}) and rc.climb (warning; live objects 1574 to 2073, actors 1). sink’s queue grew by about 500 a second and it holds the whole node’s queue, so this is critical. rc.climb is the same queue seen from the heap: messages are heap objects. The --dump file keeps both raw snapshots for the incident ticket.

3. Is it draining at all?

forge top --env prod --sort msgs_in --once -n 5

web-1  actors 1, queued 2598, rss 5 MB, busy 0%, crashes (1h) 0, deepest mailbox sink (pid 0) 2598
sorted by msgs_in over 1000 ms

     PID  NAME                 TYPE               STATUS        MBOX    MSGS_IN
       0  sink                                    waiting       2597          0

sink received nothing in the last second while its queue kept growing.

4. What is it doing?

forge observe --env prod ACTOR 0

{"pid":0,"alive":true,"actor":{"pid":0,"names":["sink"],"status":"waiting","mbox":2616,
 "held":0,"msgs_in":1,"msgs_out":0, ...},"supervisor":null,"terminal":null}

msgs_in: 1: it has taken one message off its mailbox since it started, and is waiting inside that handler, not in receive: the first handler is blocked (here, a long sleep; in production, typically a slow call or I/O). With held large instead, it would be waiting in an Actor.call. The fix is in that handler: give the slow operation a timeout, move it to a task, or put a mailbox limit on sink so the node sheds load instead of growing.

A remote shell: forge shell and forge rpc

forge shell evaluates March on a running node. Each input is compiled on your machine against the project, as forge build would compile it, into a small library. That library is signed with the deploy key and sent to the node, which loads it and runs it as a task. The node answers with the result and anything the input printed.

$ forge shell --env prod
attached at epoch 3 (:help for commands, :quit to leave)
march> Actor.list(intro)
[Pid(0), Pid(1), Pid(2)]
march> let c = Actor.whereis(intro, "counter")
c : Option(Pid(a))
march> List.range(1, 1000) limit: 3
[1, 2, 3, … 996 more]
march> println("hello")
hello
()

forge rpc --env prod 'Scheduler.live_procs()'   # one input; exit 1 if it did not run

What a node needs:

• A deploy key. A --hot-reload build with --signing-pubkey, the same key forge deploy hot signs with (~/.march/ed25519_secret.key on your machine).
• A shell policy. $MARCH_SHELL_POLICY names a file listing, one per line, the capabilities an input may use. With no file, nothing is allowed.

Inputs:

The two table commands print the table instead of a record per actor. For the rows as values, call Recon.top(intro, "mailbox", 10) (a list of TopRow).

Capabilities are pre-bound names: console (Cap(IO.Console)), clock, intro (Cap(Actor.Introspect)), debug (Cap(Actor.Debug)).

A value prints by its type, the way you would write it:

march> Json.parse("{\"a\": [1, true]}")
Ok(Object([("a", Array([Number(1.), Bool(true)]))]))
march> [{ name: "first", tags: ["a", "b", "c"] }] limit: 2
[{ name: "fi"… 3 more chars, tags: ["a", "b", … 1 more] }]

• Records, tuples and constructors print field by field, with constructor names. A type that derives Show prints as its derived show would.
• Strings inside a value print quoted and escaped (["a", "b"]). A string that is the whole result, or the one field of a top-level constructor, prints as it is: Actor.inspect_state shows Ok({ n: 42 }).
• The limit applies at every depth: each list, Array, Map and Set shows at most N elements then … n more, and each string at most N characters then … n more chars. limit: all (or :limit 0) turns it off.
• The other stdlib containers print by their elements too, with the same limit: HashMap{"a" => 1}, OrderedMap{1 => "a"}, SortedSet{1, 3}, Deque[1, 2], Queue[1, 2], RRB.Vec[1, 2], NativeArray[1.5, 2.5]. A HashMap prints in hash order; the others in their own order. RingBuf and LinearMap are linear (reading one consumes it), so they print as to_string prints them. So does a container whose type name the program reuses for a type of its own (a program Deque, say).
• A type with a hand-written Show prints through its show, cut at 16 KiB with … (n more bytes).
• A function prints <fn>; a Pid and other runtime values print as to_string prints them, as does a type whose constructors are private (ptype).

The shell generates this renderer from the input’s static type, into the input’s fragment only (bin/shell_render_gen.ml, stdlib ShellRender); the node’s code is not changed.

The node refuses an input whose compiled code uses a capability its policy does not list (** refused: policy IO.Clock). This counts capabilities the input reaches through program and library code, not only the names it uses: calling a program function that reads the clock needs IO.Clock. The fragment carries its capability list, and the node checks it against the signed request after loading the fragment, before running it.

Your checkout must match the node’s build for the code an input reaches. When a session starts, the shell compares a hash of every declaration in your source with the node’s and names the ones that differ. An input that reaches one of them is refused, with the list:

error: this input reaches code that differs from the node's build:
  evens differs

Inputs that reach only unchanged code still run. A type whose constructors are numbered differently on the node (reordered, added) is always refused, since values built here would be read back wrongly there.

Spawning the program’s actors. An input can spawn one of the program’s own actors and use it like any other:

march> let k = spawn(Counter)
k : Pid({ n : Int })
march> send(k, Bump(5))
Some(())
march> Actor.inspect_state(debug, k, 500)
Ok({ n: 5 })

The actor runs the node’s code, not a copy compiled into the input: the input calls the node’s own spawn function for that actor, so the actor answers inspect_state, shows its type name in ACTORS and Recon, and is upgraded by a hot deploy like every other instance. It belongs to the node: it keeps running after the session ends (or a deploy ends it), and a later session finds it with Actor.list or Actor.pid_from_int. Spawning needs no capability of its own, but an input is charged with the capabilities the actor’s handlers use, as for any program code it reaches. An input that could only spawn the actor by carrying its own copy of the handlers is refused instead: a node built by an older compiler, or an actor whose init arguments include a function.

--force runs a read-only input over differing code. With forge shell --force (or forge rpc --force, march --shell-force), an input that reaches differing declarations runs if it is read-only. It runs your checkout’s version of those declarations against the node’s state, so its answer is what your code computes, not necessarily what the node’s would:

$ forge shell --env prod --force
march> evens(4)
warning: --shell-force: this read-only input runs this checkout's code, which differs from the node's build:
  evens differs
[0, 2, 4]
march [skew]> send(c, Bump(List.length(evens(4))))
error: this input reaches code that differs from the node's build:
  evens differs
  --shell-force runs such an input only when it is read-only, and this one is not:
    calls march_send

Read-only is decided on the compiled input, which carries a copy of all the code it runs (it calls none of the node’s functions when forced):

• every capability its code uses is one of IO.Console, IO.Clock, IO.Random, IO.FileRead (no file writes, network, processes, tasks, Vault writes, signals or foreign code);
• it calls none of the runtime’s operations that change something without a capability: send, spawn, kill, Actor.call, replies, register, monitor, setters such as Actor.set_queue_limit, the logger, and the like;
• it does not use Actor.Debug (debug, Actor.inspect_state);
• it does not use an earlier let that may hold a closure, since that closure’s code was compiled with an earlier input and is not checked.

The node’s policy still applies on top. Only differing code is forced: an input that reaches a type, actor or protocol whose definition differs is refused even with --force, since a value of the node’s could be read with the wrong layout. Once a forced input has run, the prompt reads march [skew]>, and the node’s audit line for each forced input carries "skew":1. The operations are recognised by the names of the runtime functions the input calls, so treat --force as an operator’s tool for looking, not as a sandbox.

What happens when:

• An input panics. You see ** panic: <message>; the node and the session carry on.
• An input runs too long. It is cancelled after --timeout-ms (default 10 s, at most 30 s) and you see ** timeout. A loop that never reaches a cancellation point is reported as still running on the node.
• A deploy happens. The session ends with “the node was redeployed”. Bindings belong to the session, so they go with it.
• Anything is sent. Every input, accepted or not, is appended to the node’s audit log with "type":"shell", the signer and the source. If the log cannot be written, the input does not run (** refused: audit_unavailable).
• A request is captured. Each one is signed for its session (a random challenge the node hands out when you attach), so it runs on no other connection, no other node and not after the node restarts.

Line editing and history. On a terminal the prompt is a real line editor (lib/repl/shell_line.ml, bin/shell_tty.ml). When stdin or stdout is not a terminal (a pipe, --shell-inputs, forge rpc) none of this is active and input is read exactly as before.

A line longer than the terminal scrolls sideways instead of wrapping; a resize is picked up on the next keystroke. Unknown escape sequences (function keys, PageUp) are ignored.

Inputs are kept in ~/.march/shell_history, one per line, last 1000, loaded at start. A line equal to the one before it, a blank line and :quit are not recorded. The file can hold sensitive text (a connection string or token typed into an input is kept verbatim), so it is created 0600 in a 0700 directory. Delete it, or edit it, if an input should not outlive the session.

The terminal is in raw mode only while a line is being typed and is put back before the input runs; it is also restored on exit, an uncaught exception, and SIGINT, SIGTERM and SIGHUP.

Under the hood forge shell runs `march –shell .shell