Claim: Performance

This is one of the four claims March 1.0 makes and treats any break of as a bug. The others are deploys, capabilities and refinement types.

Status: this claim is not in force yet. Unlike the other three, none of its parts is checked by CI today. This page states what 1.0 will promise and how it will be measured, so the work to get there has a fixed target. The per-benchmark targets are not decided.

The guarantee (for 1.0)

For a fixed set of benchmarks, each release of March is measured against the previous release on the same machine, and a release that is slower than the previous one beyond a stated margin is a bug, not a trade-off. Each benchmark also has a stated target relative to another language (for example “within 2x of OCaml”), and the measured results for every tier-1 platform are published for each release, with the commit and the date. Compiled programs do not leak memory, and compiling a program takes no more than a stated time per thousand lines.

How it is measured

Same machine, A against B, never against a stored number. Each run builds two compilers, the last release and the candidate, and runs every benchmark with both, on the same idle machine, interleaved, several times, comparing medians. A millisecond figure recorded on another day is not a baseline: in July 2026 hash_map_bench read 25-29% slower than its recorded figures, yet an A/B of the old and new compilers on one machine showed the new one 13% faster; the recorded figures had been taken on a quieter machine (specs/benchmarks.md, “why absolute-ms baselines cannot detect regressions”). The runner that does this is being built (road-to-1.0 plan, Phase 2).

Counts where counts are possible. Allocation counts (live_allocs()) and string statistics (MARCH_STRING_STATS) don’t depend on load, so they can gate every pull request where timings can’t.

Benchmarks are always compiled (march --compile --opt 2), never interpreted.

Targets

Benchmark Compared with Target
bench/fib.march OCaml TBD
bench/binary_trees.march OCaml TBD
bench/tree_transform.march OCaml TBD
bench/list_ops.march OCaml, Rust TBD
bench/string_*.march TBD TBD
HTTP (bench/http_get*.march) TBD TBD
JSON (bench/json_stream*.march) TBD TBD
Actors (bench/actor_ping.march) Erlang TBD
Regression margin against the last release (same benchmark) TBD
Compile time (seconds per thousand lines) TBD

Which benchmarks are in the fixed set, which languages they are compared with, and what ratios they must reach is an open decision (road-to-1.0 plan, open decision 7). The rows above are candidates, not commitments.

Assumptions

  • Results hold for the tier-1 platforms (darwin-arm64, linux-x86_64, linux-aarch64) on the machines they were measured on. Other hardware gets other numbers; the ratios are what carries over.
  • The machine is idle while measuring. A loaded machine makes every benchmark slower and noisier.

Not claimed

  • Any number measured on your machine. The published figures are a record of the benchmark machine.
  • Programs outside the fixed set, or the interpreter and REPL.
  • Freedom from leaks today. Several leaks are open, for example specs/todos/2026-10-12-lsan-sweep-remaining-leaks.md and specs/todos/2026-10-12-unused-owned-closure-param-leak.md.
  • Platforms below tier 1, the browser playground, or WASM.

Gate

Runs in CI today:

  • bench-gate compiles every registered bench/*.march at --opt 2, runs it and checks the value it prints. This proves the benchmarks still compile and compute the right answer. It does not measure time.

Planned, not built:

  • The same-machine A/B runner against the last release (being built), run nightly on a dedicated quiet machine.
  • A leak gate: allocations still live at exit must stay flat across a fixed corpus.
  • Allocation-count gates on every pull request.
  • A compile-time budget test that catches a compile that grows faster than the program.
  • A published, dated results page per release.