Parser
Parser module: parser combinator core.
Design: specs/plans/2026-08-09-parsing-and-string-search.md
Two commitments shape every type in here:
- Byte offsets, not code points. Position is an Int index into the
input string, and input is inspected with `string_byte_at`, which
returns 0..255 (or -1 past the end) and allocates nothing. March has
no character literal, so the byte primitive takes an Int code and the
ergonomic primitive is `lit`, which takes a String.
2. **Errors are the product, so the success path must not pay for them.**
`ROk` carries only integers alongside the value — in particular the
furthest offset any sub-parse failed at, which `alt` needs in order to
report the alternative that got deepest rather than the one listed
last. Expected-sets and context stacks are built only on failure.Failure is split into two constructors rather than a boolean flag, so that ordered choice cannot forget to check it:
RFail soft — `alt` may try the next alternative
RCut hard — a commit point was passed; propagate, do not backtrackPrimitives: lit, byte, byte_if, take_while, take_while1, eof, pure Sequencing: and_then, skip_then, skip_first, map, flat_map, delay Optionality: optional Choice: alt Repetition: many, many1, repeat, sep_by, sep_by1 Error control: commit, then_commit, ctx, label, fence, recover Running: run_all (whole input), run (prefix) Rendering: render, line_col
Start with run_all: run succeeds on a valid PREFIX and drops the rest, which is the usual way a combinator grammar silently gives a wrong answer.
Still to come: an OCaml-side renderer into lib/errors so library and compiler diagnostics are indistinguishable.
Types
Functions
Ordered choice: try p, and only if it fails softly try q.
A hard failure from `p` — one raised past a commit point — is propagated
rather than swallowed, which is the whole point of committing. When both
alternatives fail softly the two errors are merged by `merge_err`, so the
reported position is the furthest either one reached.Run p, then q, and pair their results.
A failure in `q` is reported at `q`'s own offset, so the error points at
where the input actually diverged rather than at the start of the sequence.
A hard failure (`RCut`) from either side propagates without being turned
into a soft failure — that is what makes a commit point stick.Match one byte by its code (0..255).
There is no character literal in March, so byte codes are the primitive and
`lit` is the ergonomic form for anything spelled as text.Match one byte satisfying pred, described to the user as name.
Named `byte_if` rather than the conventional `satisfy`: `satisfy` is a
reserved keyword in March (it belongs to refinement types), so `fn satisfy`
does not parse. `byte_if` also pairs with `byte` and says what it does.
The name is what the reader sees in a message ("I was expecting a digit"),
so it should name the class, not the predicate: "digit", not "is_digit".
`string_byte_at` returns -1 past the end of the input, and a predicate over
byte values will reject that, so end-of-input needs no special case.Commit: past this point, failure is a hard parse error rather than a backtrack into the next alternative.
This is SNOBOL's `FENCE` and Prolog's cut. Without it a malformed `if`
statement silently falls through to be parsed as something else, and the
error message ends up pointing at the wrong construct entirely.
A success is passed through untouched; only a soft failure is upgraded.Name the construct a failure happened inside.
On a hard failure, pushes `(name, start_offset)` onto the error's context
stack. That stack is what turns
error at 14:3: I was expecting `end`
into
error at 14:3: I was expecting `end` to close the `block` that started at 12:1
Only hard failures carry context: a soft failure is a path not taken, and
naming every alternative a parser merely tried would be noise, not context.Defer building a parser until it is run.
Combinators are ordinary VALUES, built eagerly, so a rule that mentions
itself would recurse while being *constructed* — before a byte of input is
read — and never return. Every recursive or mutually recursive grammar
therefore needs its back-edge wrapped:
pfn value() : Parser(Json) do
alt(number(), array_of(delay(fn -> value())))
end
This is not a March quirk; every combinator library in a strict language
needs the same device (Scala's `P(...)`, OCaml's `fix`). It is the one piece
of ceremony a grammar DSL would remove, since a DSL can see the recursion.Succeed only at the end of the input.
Without this a grammar silently accepts a valid prefix of invalid input —
`1 + 2 )))` parses as `1 + 2` and the trailing garbage is never reported.
At the top level you rarely need to write this yourself: `run_all` is
`run` with `eof` already appended. Reach for `eof` directly when the
end-of-input requirement sits INSIDE a grammar — one branch of an `alt`,
say — rather than at the outermost call.Scope a commit point.
Runs `p`, but converts a hard failure back into a soft one at this
boundary, so one construct's `commit` cannot abort a sibling. This is the
other half of SNOBOL's `FENCE`: committing is scoped, not global.Sequence, where what to parse next DEPENDS on what was just parsed.
This is the one combinator `map`, `and_then` and `alt` cannot replace
between them. They all fix the shape of the rest of the parse in advance;
`flat_map` chooses it from a value. Length-prefixed data, indentation
sensitivity, and version negotiation all need exactly this — which is the
formal difference between applicative and monadic sequencing, and the reason
a grammar DSL built only on `/`, `*` and `?` cannot express them.
-- a digit says how many items follow
flat_map(digit(), fn n -> repeat(item(), n))
The furthest-failure position is threaded through both halves, so an error
in the dependent parser still competes correctly inside an `alt`.Replace a parser's expected-set with one human-named class.
Raw expected-sets degrade into token soup — "expected `-`, `0`..`9`, `(`, or
`fn`" — so every public nonterminal in a grammar should be labelled.
The rule that keeps labels honest: the substitution happens ONLY when `p`
failed **without consuming input**. If `p` got somewhere before failing, its
inner error is more specific than the label and is kept, so a label can
never hide real progress. Consumption is judged by comparing the failure
offset against the offset the parser started at.
A hard failure is never relabelled either: past a commit point the specific
error is the whole point.
run(label("a number", lit("0")), "x") -> Err(..)
-- 1:1: I was expecting a numberConvert a byte offset into a 1-based (line, column) pair.
Match a literal string at the current offset.
Compares byte by byte with `string_byte_at`, so nothing is allocated while
scanning and a mismatch past the end of the input behaves like any other
mismatch (`string_byte_at` returns -1 there, which never equals a byte of
the pattern).
On failure the reported offset is where the literal *started*, not where the
bytes diverged: a caller who wrote `lit("ab")` wants to be told that `ab`
was expected here, not that `b` was expected one byte along.Zero or more occurrences of p, in order.
Stops at the first soft failure and succeeds with what it has. A HARD
failure propagates instead: once a commit point inside an item has been
passed, the item was not optional, and quietly ending the list there would
turn a real error into a short parse.Transform a parser's result, leaving its failure behaviour untouched.
Try p; on a soft failure succeed with None, consuming nothing.
A hard failure still propagates: past a commit point the construct was not
optional.Turn a failure into a value and resynchronize.
If `p` succeeds the result is `Ok(value)`. If it fails — softly or hard —
the error becomes `Err(err)`, the input is skipped forward past the next
`sync` match, and the parser SUCCEEDS. Nothing above it sees a failure, so
a driver can keep going and collect the rest.
`many(recover(item, sync))` is the intended shape: it returns every item
that parsed and every error that did not, from one pass.
**Have the item consume its own separator.** Otherwise the parser is
handed a separator where an item is expected, fails there too, and you get
a spurious error per separator. On `"ok;BAD;ok;ALSOBAD"`:
many(recover(lit("ok"), lit(";"))) -- 4 errors
many(recover(lit("ok") |> skip_then(opt_semi), sep)) -- 2 errors
Both are "correct"; only the second reports what a reader would call the
mistakes. Resync skips *past* the sync match, so a sync token the item
does not own becomes the next thing tried.
Note the interaction with `many`'s zero-width guard. At end of input `p`
fails and `skip_to` cannot advance, so `recover` succeeds without
consuming and `many` stops — which is the termination argument. Without
that guard this pair would spin.Render a failure as a diagnostic in the March compiler's voice.
2:3: I was expecting `Z` in the block that started at 1:1
Only the innermost context frame is named. A full stack reads as a trace,
and a parse error is not a stack trace — the construct you are inside is
what locates the mistake.Exactly n occurrences of p, failing if there are fewer.
Unlike `many`, a shortfall is an error rather than a shorter list: `n` came
from somewhere — usually a length prefix — so falling short means the input
disagrees with itself.Run a parser over an input string, consuming as much as it matches.
Both a soft and a hard failure surface as `Err` — the distinction between
them matters to `alt` while parsing, not to the caller of `run`.
**`run` succeeds on a valid PREFIX and ignores the rest.** That is the
single most common way to get a wrong answer out of a combinator library:
run(digits(), "123xyz") -> Ok("123")
-- the trailing `xyz` is never looked at
That behaviour is correct for a parser you are composing INTO a larger one,
and wrong for a top-level "parse this whole input" call. For the latter
reach for `run_all`, which is this plus `eof`.Run a parser over an input string and require it to consume ALL of it.
This is `run` with `eof` appended, and it is what you almost always want at
the TOP level of a grammar: it turns "parsed a prefix, silently dropped the
rest" into a real error that points at the first byte the grammar could not
account for.
run_all(digits(), "123xyz") -> Err(..)
-- 1:4: I was expecting end of input
run_all(digits(), "123") -> Ok("123")
Use `run` instead only when a leftover tail is meaningful — streaming, or
hand-managing the offset yourself.Zero or more occurrences of p separated by sep, discarding separators.
One or more occurrences of p separated by sep.
Run p, then q, and keep q's value — the mirror of skip_then.
Run p, then q, and keep p's value.
This is the pipe-friendly sequencing form: `a |> skip_then(sep)` reads as
"a, then a separator I do not care about". Pairing with `and_then` gives the
NimbleParsec-style pipeline the design notes describe — and the left-nested
tuples that make the case for named binders past three elements.Match a run of zero or more bytes satisfying pred, returned as one String.
Never fails, so it is the right shape for optional whitespace.Match a run of one or more bytes satisfying pred, returned as one String.
The reason this exists rather than `many(byte_if(...))`: that spelling
allocates a list cell per byte and then needs a second pass to rebuild a
string. This scans to the end of the run and takes a single slice — one
allocation per token instead of one per character, which is the difference
between a combinator lexer being usable and being a toy.Run p, then commit to q — the correct shape for "past this token, the construct is no longer optional".
**Where a commit sits decides whether a later failure is reported at all**,
and getting it wrong is silent. `commit(delimiter)` makes only the
DELIMITER's own failure hard; everything sequenced after it stays soft, so
an enclosing `sep_by` or `alt` quietly backtracks and replaces the real
error with a shallow one:
-- WRONG: commit on the ":" only
and_then(key, and_then(commit(lit(":")), value))
"{a:}" -> 1:2: I was expecting `}`
-- RIGHT: commit on everything after the key
then_commit(key, skip_first(lit(":"), value))
"{a:}" -> 1:4: I was expecting a value in the member that started at 1:2
Both accept `{}`, so the difference never shows up on valid input — only on
the malformed input where the message mattered. This combinator exists so
the right shape is the short one.