Benchmarks¶

fnloom against Apache Hamilton 1.90.0 on generated graphs, by wall clock. fnloom is meant to be faster at building a graph and at running a prepared plan, so a row where it's reliably slower counts as a bug.

At a glance¶

Results¶

Milliseconds, Hamilton / fnloom with input type checks on / fnloom with them off; lower is better. Hamilton has no prepared plans and no way to switch its input checks off, so it is measured as it ships. The last column is fnloom's alone (checks on / off). A cell marked † is one where fnloom with checks on was slower than Hamilton in that single sample.

Measured on an Apple M2 Pro with 16 GB, macOS 27.0, apache-hamilton==1.90.0, and a release build of the extension. The chain, layered and rows rows were measured on 2026-10-27, with the interpreter recorded then as CPython 3.14.9; the extract and types rows on 2026-10-28 with CPython 3.14.8. The fan_in row was measured again on 2026-10-28 with CPython 3.14.8, all three runtimes in one session (python -m benches.run fan_in).

extract is one @extract_fields producer returning a dict of n int fields, each read by its own consumer, with every consumer requested. fnloom checks each field against its type on every run, with input checks on or off (Q61), and still runs 8.6 to 998 times faster than Hamilton's execute, which checks none. Hamilton's cost grows faster than n: 65.8 ms at 1,000 fields, and 6.4 s at 10,000.

types is fan_in over typed inputs: one function whose n parameters are declared, in turn, Optional[int] (every other one given None), tuple[int, str, float] and list[int]. fnloom checks each tuple's length and items (Q80), and None against the type (Q79). Hamilton refuses every value of a fixed-length tuple type, so its module declares those parameters as a bare tuple, which it checks with one isinstance. Each type's check is worked out once, when the plan is prepared: Optional[int] is one isinstance against (int, NoneType), a list for list[int] passes on its class, and a tuple checks its three items. Steady execution is 2.1 to 13 times faster than Hamilton's execute, the least at 10,000 parameters, where, as for fan_in, the one call with 10,000 keyword arguments is most of the cost. The checks cost about 0.18 ms per 1,000 inputs.

Deep checks and widening are opt-ins (see Deep checks and numeric widening). The types shape was timed with each on, checks on, alongside the default, as runtimes fnloom (deep) and fnloom (widen) of benches.run (medians of three runs, ms, on 2026-10-28; default / deep / widening):

The default stays within noise of the table above. Deep checks cost about 0.05 ms per 1,000 inputs here, each list holding one item: a list[int] of exactly the class checks its items in a loop of isinstance calls, made when the plan is prepared. The cost grows with the items read: a list[float] of 100,000 items takes about 2.6 ms, and a list[tuple[int, str]] of 100,000 about 31 ms. An item that is itself a collection of collections, or meets a union or an alias, is walked question by question and costs more per item (over 200,000 items each):

A check that fits reads each part the value shares once, even on a cycle, never once per path; a check that fails may read shared parts more than once. Widening costs nothing on a value that fits without it.

Steady execution, the cost of running a prepared plan again, is where fnloom differs most. With checks on, it's 2.0 to 24 times faster than Hamilton's execute on the first four shapes wherever both finish, and 8.6 to 998 times on extract, because the plan is compiled once and each run is a loop over its steps in Rust. The low end is fan_in at 10,000 parameters (2.0×), where one Python call with 10,000 keyword arguments is most of the cost for both.

Input checks are close to free. The gap between the two fnloom figures is within noise: steady execution with checks on and off differs by at most 0.04 ms on every graph of the first five shapes but two at 10,000: fan_in, 21.8 against 21.3 ms (2%), and extract, 6.385 against 6.304 ms (1%). A check against a plain class such as int, or a union of them such as int | None, is compiled once per plan into one isinstance call. A generic or a fixed tuple gets a check made once per plan too, and only a value it can't settle by class takes the general path.

Where Hamilton raises¶

Hamilton builds the 1,000- and 10,000-node chains, the 10,000-node layered graph and the 1,000- and 10,000-row rows graphs, then raises RecursionError on the first execute: its graph walk recurses once per level of depth. fnloom runs all 24 cases, with checks on and off, and every answer it gives is checked against the shape's closed form.

Build: re-measured¶

The design calls a row where fnloom is reliably slower a bug. The table's build column is one sample per cell, with Hamilton running first in the same process, so it cannot settle this. The build it marks †, and the fan_in builds that were reliably slower before (#5, #6, #7), were re-measured with one fresh process per sample, nine samples per runtime and case, the two runtimes alternating which goes first, on a release build, timing only the build call (medians, ms, Hamilton / fnloom):

The fan_in builds the table marks † after its 2026-10-28 re-measure were measured the same way that day, with CPython 3.14.8:

No build is reliably slower. The checks-on and checks-off builds do the same build work, since input checks run per run, so their differences in the table (for example rows 10,000, 128.1 against 60.9 ms) are noise.

A build was slow from per-parameter and per-node Python work, not from the Rust core, which is under a fifth of it. For an ordinary function the scan reads its parameters off its code object and its kind off its code flags, keeping inspect for wrapped functions, a __signature__ and the unsupported kinds; resolving skips typing.get_type_hints when every annotation is already a class, keeps every parameter and node it would make again unchanged, and skips plain classes when it looks for type aliases; an input read by one parameter is typed without comparing consumers; an undocumented module-level function skips the search for an inherited docstring; and an input's NodeView is made when Driver.nodes first reads it, while the input checks read each input's type without making its view.

Building from functions¶

Builder.with_functions builds the same graph as with_modules from the module's functions given one by one (Q72). benches/builds.py times both, and Hamilton's nearest way, ad_hoc_utils.create_temporary_module and a build of it, on a release build, median of nine builds each, interleaved in one process (ms):

Measured on the machine above, on 2026-10-28. Giving the functions costs under a tenth more than scanning the module: each function is checked once, when it is given, and then scanned as a module's function is.

Building with namespaces¶

benches/builds.py also builds the 1,000-node layered graph two ways: as one module, and as ten modules of 100 nodes, each under its own namespace, ns0 to ns9, all reading the one input seed (Q74). Same machine and release build, median of nine builds each, interleaved in one process (ms):

Measured on 2026-10-28. Namespaces cost under a tenth more: every parameter of a namespaced node reads a node of its namespace under a dotted name, so the core passes it under the parameter's own keyword, one rename per edge. The namespace's names are found once, before any node is made, so each parameter is bound in the same pass as without one.

Per run¶

A prepared plan run once per incoming record (a market bar, say) pays its per-run overhead every time. Microseconds per PreparedPlan.run on the same machine, median of nine batches, before and after the per-run work, checks on:

What made the difference: input checks compiled to isinstance for plain classes; inputs read and outputs keyed by each node's interned name, so a run makes no name strings (Plan::run_by_id in the core); a @parameterize row whose sources each feed one parameter called directly, passing each source under its parameter's name, with its value() constants bound by functools.partial, instead of through a Python wrapper; and Driver.execute skipping the plan cache's key and lock when a request repeats the last one, the common case in a loop. One node call costs about 130 ns in the native run, against about 30 ns for a bare Python call; a run's fixed cost is about 0.3 µs native and 0.7 µs through PreparedPlan.run, and Driver.execute adds about 0.3 µs more.

Caching¶

Cached runs against uncached ones and against Hamilton 1.90.0's with_cache, on a 200-node chain and a 1,000-node layered graph of trivial nodes, the worst case for a cache: a node costs less to run than to look up. Milliseconds, cold / warm, medians over seven fresh processes, each the median of three cold runs (a new driver with an empty store) and the 21 warm runs after them. Uncached, cold is a first run and warm a steady one. fnloom runs a prepared plan; Hamilton, which has none, runs execute. So fnloom's cold and warm times leave out prepare, made once before the timed runs, while Hamilton's include the planning each execute does. Hamilton's memory store is its InMemoryMetadataStore with its InMemoryResultStore, and its disk store the default with_cache(path=...).

Measured on the machine and build above, on 2026-10-28. A warm fnloom run is 260 to 1,070 times faster than Hamilton's on the same kind of store, and 7 to 43 times faster than Hamilton's uncached run. A warm fnloom run on MemoryCache is faster than an uncached fnloom run: it loads the outputs and runs nothing. Hamilton's warm runs cost more than its uncached ones, 25 to 27 times as much in memory, as the caching design found (Q16).

fnloom, async is AsyncDriver with sync_nodes="inline", timed on the loop; Hamilton's async driver has no row, since it fails to build with a cache (Q68). Its warm runs load the outputs and run nothing, as the sync driver's do. Its cold runs with MemoryCache cost 3.7 to 3.8 times its uncached ones, against 1.9 to 2.3 times on the sync driver. Two things make that up. The async walk is Python, so it looks up and stores each node through Python calls instead of the native walk a plain MemoryCache gets on Driver. And a cached async plan builds its Schedule lazily, on its first run, so the cold figure counts it, where the uncached figure leaves it to prepare. With DiskCache, the files cost the same on either driver.

The targets of the caching design (§8), with fnloom's figures:

The cold target is now per node. It was "at most 1.5 times an uncached run", and a cold run cost 5.99× and 5.45× on the same day before #10. A cold run now pays about 0.19 µs per node over an uncached one, where an uncached trivial node costs about 0.16 µs. Most of that is the key, one BLAKE3 compression per 64 bytes read, about 75 ns, which alone is half of a trivial node, so no ratio near 1.5 is reachable on trivial nodes without a weaker key. The rest is a lookup and a store in the native memory store. The cost is per node, not per byte, so the design now bounds it per node: at most 0.25 µs.

What closed most of the gap:

• The walk and the memory store are native. A plain MemoryCache keeps its entries in the extension. The walk back from the outputs looks keys up there, and the run stores each result as its node succeeds, with no call into Python per node. A subclass of MemoryCache, or any other store, takes the Python path.
• A key is one BLAKE3 block where it can be. A key is BLAKE3 keyed by the node's static part, over the 32-byte versions of what the node reads. The parameter names and value() constants that used to be framed into every key are hashed once, at build, into the static part, so a node reading one value hashes 32 bytes.

Async¶

AsyncDriver against Hamilton 1.90.0's async_driver.Builder, whose result is a dict, on the chain and layered graphs of trivial nodes, once with every node an async def and once with every node a plain function. fnloom runs in both sync_nodes modes. Milliseconds, first / steady: a first run on a new driver, then the median of the 21 runs after it, each row the median over five fresh processes. fnloom runs a prepared plan, so its first run leaves out prepare and the schedule it builds from the plan's steps, both made once before the timed runs. Hamilton, which has no prepared plans, runs execute, planning included.

Measured on the machine above, with CPython 3.14.8 and a release build, on 2026-10-28. Wherever both run nodes on the loop, which means every async row and the inline column, fnloom is 20 to 84 times faster than Hamilton on a first run and 24 to 90 times faster on a steady one. The smallest gain is on the 100-node layered graph of async def nodes, 19.7× on the first run in either column. The generated async def nodes never suspend. fnloom starts each node's task eagerly, so a node that doesn't suspend finishes with no turn of the loop, while Hamilton schedules a task for every node. Without a custom task factory on the loop, fnloom makes each task directly instead of through loop.create_task, which took about a tenth off the 1,000-node async def rows (steady, 0.882 to 0.785 ms on the chain and 1.013 to 0.911 ms on the layered graph, against the previous run). A node that awaits real I/O pays for its suspensions on either.

thread is slower than Hamilton on the layered graph of sync nodes, by design. Each sync node makes a round trip to a worker thread: loop.run_in_executor queues the call to the loop's default executor, a worker thread wakes to run it, and the loop wakes again for the result. That costs about 34 µs per node on the chain, where the nodes run one after another, and about 23 µs on the layered graph, where ten at a time are in flight. A trivial node costs well under 1 µs, so the round trip is the whole cost. On the chain, thread is still 1.4 to 1.6 times faster than Hamilton. These are the rows where it's slower:

Hamilton runs sync nodes on the loop and stops it while each one runs. The thread is what keeps the loop free (Q65), so the thread column isn't a like-for-like comparison. For a graph of cheap sync nodes, sync_nodes="inline" is the remedy, and it's the fastest column. A dedicated executor and dropping the per-call context copy were tried. Each saved 1 to 2% of the round trip, within noise, so neither was kept.

Hooks¶

A hook costs a run nothing until one is attached: a driver with no hooks, output checks, value checks or result builder runs the code it ran before hooks existed. With one attached, every node a run reports makes two events and two Python calls. The hook timed here overrides all four methods and only counts the events (benches/hooks.py), so what it measures is the hooks' own cost.

Unhooked runs are unchanged. benches.run and benches.async_ were run on the commit before hooks (1374b99) and with them, one after the other on the same machine and release build. Steady execution moved by 0.94× to 1.06× across all 56 fnloom rows of benches.run, a median of 1.00×, and every async row by 0.99× to 1.02×: within noise. At 1,000 nodes, steady execute with input checks on (ms, before / after):

A hook costs about 2 µs per node, on both drivers. Steady execution with the hook, minus without it, divided by the nodes the run reports: 1.92 to 1.98 µs per node on the 1,000-node chain, layered and rows graphs and on extract (whose 1,000 fields report as producer, fields and consumers: 2,001 nodes). A run adds about 5 µs of its own, for before_run and after_run: fan_in and types, one node each, pay 6.5 to 9 µs a run, which is within noise at 10,000 parameters. On AsyncDriver, benches.async_ gives 1.76 to 1.84 µs per node on the loop, in either sync_nodes mode for async def nodes and inline for plain ones. With sync_nodes="thread" it is 2.3 to 3.3 µs per node, since the events are made on the loop's thread around each round trip to the worker.

For comparison, an unhooked node costs about 0.13 µs, so a hooked run of trivial nodes is about 15 times slower than an unhooked one. Hamilton's own run costs about 2.5 to 3.5 µs per node with no adapter at all.

A hooked run is still faster than an unhooked Hamilton run. benches.run, with input checks on, milliseconds, Hamilton / fnloom hooked / fnloom unhooked; the last column is the hook's cost. A cell marked ‡ is one where hooked fnloom was slower than Hamilton in that single sample:

Measured on the machine above, with CPython 3.14.8 and a release build, on 2026-10-28. Steady execution with the hook is faster than Hamilton's execute on every graph, by 1.22× (chain, 100) to 135× (extract, 10,000), and it runs the graphs where Hamilton raises. The narrow margin is on chains of trivial nodes, where both costs are per node and fnloom's hooked node, about 2.1 µs, is close to Hamilton's unhooked one, about 2.5 µs.

The first execute of a small graph is slower with a hook. On the seven cells marked ‡, all at 10 or 100 nodes, a hooked fnloom's first execute took 3% to 43% longer than Hamilton's, one sample each. The first execute compiles the plan, and a hooked plan also makes each reported node's view and its input re-keying, once per plan; a prepared plan, or a second execute of the same request, pays none of it again. Unhooked, fnloom's first execute is faster than Hamilton's on every graph.

On AsyncDriver, a hooked run is faster than Hamilton's async driver on every row of benches.async_, by 1.5× to 23×, except where the unhooked thread mode already is not (Async): the layered graph of plain functions with sync_nodes="thread", at 2.652 against 2.140 ms (100 nodes) and 26.0 against 22.3 ms (1,000), steady. With inline, the remedy there, the same rows take 0.253 and 2.517 ms.

With a cache. A hooked run on MemoryCache walks the keys in Python, not natively, since the hooks need each node's outcome, so it pays more than the hook alone. benches.cache, runtime fnloom-hooked (ms, cold / warm, medians over seven fresh processes):

A hooked cold run costs about 1.0 µs per node more than a hooked uncached one (the Python walk, the keys and the stores), and about 2.8 µs per node more than an unhooked cold run. A hooked warm run loads the requested outputs and reports them as hits: 0.056 ms for the chain, one hit, and 0.246 ms for the layered graph, ten hits. That is 0.04 to 0.16 ms more than unhooked, the cost of the Python walk; making the native walk return its keys would remove it. A hooked cached run is still 25 to 360 times faster than Hamilton's in memory.

Checks and wrappers¶

An output check or a node wrapper costs a run nothing until one is added: a graph with neither builds the same payloads it did before Plan 8. With one, each checked or wrapped node makes one more Python call around its own. On a 1,000-node chain, prepared once and run 200 times (the median of each, ms, the development build of scripts/check.sh, so slower than a release build in absolute terms):

benchmarks/test_checks_wrappers.py measures the checked and the wrapped runs under CodSpeed.

Cold start, first call and memory¶

The tables above time a warm process. A short-lived process (a CLI, a job, a serverless handler) pays for the import, the build and the first call before any warm run, and holds the libraries in memory the whole time. benches/realworld.py measures those on a workload shaped like an application's, in a fresh interpreter per sample, as the median of seven, the runtimes round-robin. A sample imports the library, imports the module, builds a driver, makes one execute, then times 15 more and reads the process's peak and final resident set size (RSS). The answers of the two runtimes are checked against each other.

The workloads are written to files in each runtime's spelling:

• mixed: 44 nodes of numpy arrays (20,000 floats): a chain of transforms, ten @parameterize scales, an @extract_fields of five statistics, ten @parameterize energies and a report that reads 20 of them.
• mixed_async: the same, with the chain and the report async def. The @parameterize rows stay plain functions, because Hamilton's async driver passes a parameterized async def node's dependents a coroutine instead of its value.
• rows_np: one function of 200 @parameterize rows, each a numpy dot product, all 200 requested.

Numpy is lazy (each node imports it when it first runs, as an application with optional heavy dependencies does) or loaded (imported, and its random and fft submodules used, before the timed part). Hamilton is run with its extensions autoloaded, as it ships; no extension library is installed here, so that run costs about the same as with HAMILTON_AUTOLOAD_EXTENSIONS=0 (hamilton-noext in the raw rows).

Import and first build, loaded. Milliseconds, median of seven processes:

Import is import fnloom against import hamilton plus hamilton.driver and hamilton.async_driver. fnloom imports 6.1 to 6.4 times faster here. A downstream consumer's migration measured about 1.42 s for Hamilton's import and 21 ms for fnloom's, and a cold import-plus-build summed over 30 graphs at 3.9× faster; the import gap grows with the extensions installed (Hamilton imports every one it finds), and the build gap with the graph. On 44 nodes the builds tie, at about a millisecond each. Only 200 @parameterize rows separate them, 1.6× in fnloom's favour.

Memory. RSS in megabytes after building and running, median of seven. A bare interpreter is 27.4 with the lazy numpy workload and 40.4 with numpy imported:

fnloom's process is 1.8 to 2.2 times smaller: it adds about 10 MB to the numpy baseline and Hamilton about 50 MB (the async driver, about 73). The difference is Hamilton's import, not the graph: the numbers barely move between the sync workloads and rows_np.

First call against steady state. Milliseconds, first / steady (the median of 15 warm execute calls):

Two things are in these rows.

1. The lazy first call is slower in fnloom, and that is numpy, not fnloom. import hamilton itself imports numpy (and numpy.random), so a node's import numpy is already free by the first call. fnloom imports neither, so the first node to run pays for them: the cost moves from the import to the first call. Profiling the first execute of rows_np with numpy loaded, all of it is numpy.random's extension modules (11 ms), none of it fnloom (driver.execute, prepare and _compile are each under 0.5 ms). Added to the timeline, fnloom reaches the first result in 82 ms (35.7 + 0.3 + 1.0 + 45.5, mixed) and Hamilton in 258 ms (252.4 + 0.4 + 0.9 + 4.6). A process that already imported the heavy libraries when it started, or one that wants a flat first call, imports them at startup, and then the first and steady calls are within a millisecond of each other.
2. One real first-call loss: mixed_async, 5.16 ms against 3.73 ms (1.4× slower), with sync_nodes="thread". The graph's sync nodes make the first call start the loop's default executor and a worker thread: with sync_nodes="inline" the same first call takes 2.9 ms, against 5.1 ms for thread, and warm calls 1.9 against 2.5 ms. About 1.6 ms is thread start-up, once per event loop, and about 1 ms the plan compile. Neither is hashing or type-check setup: the profile of a first call shows no code-version hash of a helper, a constant or a library, and a first call of the sync workloads is within 0.5 ms of a warm one, the plan compile. Nothing here is a bug to fix, so no change was made; inline removes the thread start for graphs of cheap sync nodes, as it removes the per-node round trip described under Async.

Why fan-in is fast¶

A node with n parameters is called with n keyword arguments. CPython binds a keyword to a parameter by identity with the interned parameter name, and when no name matches by identity it compares the keyword with == against every parameter, so fresh name strings make a call cost n² string comparisons. fnloom interns every node's name once, when the graph is built, and passes those exact strings (tests/test_binding.py::test_keyword_names_reach_the_function_interned). What remains is CPython's own identity scan, n pointer comparisons per keyword, which Hamilton's calls pay too.

Run it¶

uv run maturin develop --release   # time an optimised extension, not the debug build
uv run python -m benches.run > bench.jsonl 2> bench.stderr
uv run python -m benches.run extract > extract.jsonl 2> bench.stderr   # only the shapes named

It prints 104 JSON lines to stdout (shape, n, runtime, input_checks, build_ms, first_execute_ms, steady_execute_ms, prepare_ms, error, error_phase). Hamilton prints a help banner to stderr whenever it raises, hence the second redirect. Each runtime's answer is checked against the shape's closed form, and a mismatch aborts the run. Generated modules are dropped from sys.modules after each case.

The cached runs have their own script, which starts a fresh process per sample:

uv run python -m benches.cache > cache.jsonl

It prints 24 JSON lines (shape, n, runtime, store, cold_ms, warm_ms, warm_loads, processes, error), one per shape, runtime (fnloom, fnloom-hooked, fnloom-async or hamilton) and store (none, memory or disk). fnloom-hooked is fnloom with the counting hook. warm_loads is how many entries a warm fnloom run loads, counted on a separate, untimed driver. The graphs are written to files and imported, since Hamilton's cache versions a function by its source, and every answer is checked against the shape's closed form.

The cold-start, first-call and memory runs have their own script, also a fresh process per sample:

uv run python -m benches.realworld > realworld.jsonl

It prints 20 JSON lines (workload, numpy, runtime, processes, import_ms, module_ms, build_ms, first_ms, steady_ms, peak_mb, rss_mb), one per workload, numpy mode (lazy or loaded) and runtime (fnloom, hamilton or hamilton-noext), plus two none rows for a bare interpreter. See Cold start, first call and memory.

The async runs have their own script too, also a fresh process per sample:

uv run python -m benches.async_ > async.jsonl

It prints 60 JSON lines (shape, n, nodes, runtime, first_ms, run_ms, processes, error), one per shape, kind of node (async or sync) and runtime (hamilton, fnloom-thread, fnloom-inline, fnloom-thread-hooked or fnloom-inline-hooked). Every answer is checked against the shape's closed form.

What is timed¶

All times use time.perf_counter_ns, in milliseconds. Graph generation is not timed.

• build_ms (one sample): module to ready driver (fnloom.build; hamilton.driver.Builder().build()).
• first_execute_ms (one sample): the first execute call, which for fnloom includes compiling the plan.
• steady_execute_ms: the median of 7 runs after one warm run. fnloom uses a prepared plan's run; Hamilton has no prepared plans, so it uses execute.
• prepare_ms (one sample): a cold Driver.prepare for the same request, on a newly built driver so no plan is cached. Always null for Hamilton.
• A runtime that raises records the exception class in error and the phase in error_phase (build, first_execute, steady_execute or prepare). Timings of earlier phases are kept; the rest are null.
• input_checks: Hamilton is always true, as it ships. fnloom is measured twice per graph, with the checks on and with Builder.without_input_type_checks(), and a third time as fnloom (hooked), checks on, with a hook that overrides all four methods and counts the events (benches/hooks.py).
• Argument copies ([*outputs], dict(inputs)) are made once, outside the timed regions.

Shapes¶

• chain: n nodes in a line, each adding one to the one before; one input, seed.
• fan_in: one function with n parameters, all inputs, summed.
• layered: n nodes in rows of ten, each reading two nodes of the row above; one input, seed.
• rows: one function with n @parameterize rows, each sourcing the row before it, the first sourcing the input seed. The module is generated in Hamilton's spelling for Hamilton and in fnloom's for fnloom.
• extract: one @extract_fields producer returning a dict of n int fields, each read by its own consumer.
• types: one function with n input parameters, declared in turn Optional[int], tuple[int, str, float] and list[int], summing the int in each. Hamilton's module declares the tuples as a bare tuple, since it refuses every value of a fixed-length tuple type.

The instruction-count gate¶

Wall-clock time is too noisy on shared CI runners to gate on, so CI gates the Rust core instead: it counts the instructions Graph::build, Graph::prepare and Plan::run execute under callgrind, on chain, fan-in and layered graphs of 1,000 and 10,000 nodes (crates/fnloom-core/benches/core.rs). A rise of more than 2% over the committed crates/fnloom-core/benches/baseline.json fails the instruction-counts job, which runs scripts/bench_gate.py.

Counts depend on the CPU architecture, the toolchain and the C library, so the baseline is generated in the job's own environment: the rust:1.99.1 image on aarch64 Linux. A change that costs instructions on purpose regenerates it with --write-baseline and says why in its commit message.

The CodSpeed gate¶

The Python run path is measured by CodSpeed as well. benchmarks/ holds pytest benchmarks of a prepared plan's steady run on each generated shape, Driver.execute on a repeated request, cached runs against a MemoryCache (warm and cold), the async driver in both sync_nodes modes, @extract_fields, @extract_columns and @config, the mixed numpy workload of the cold-start section (a steady run, two builds and the async driver in both modes), hooked runs of a 1,000-node chain on both drivers (with and without output checks, and cached, warm and cold), and Builder.build on a 1,000-node graph. The extension is built in release mode first, so the numbers are those of the optimised build.

CodSpeed counts simulated CPU instructions and allocations instead of reading a clock, so a result is stable on a shared runner, and each pull request is compared against main. It complements the tables above, which are wall clock on one machine and show how fnloom compares with Hamilton, and the callgrind gate above, which covers only the Rust core: CodSpeed sees the whole call from Python, the scan, the plan cache and the node calls included.

To enable it, install the CodSpeed GitHub app for the repository and set the repository variable CODSPEED_ENABLED=true. Until then the benchmarks job is skipped, and the benchmarks-smoke job still runs every benchmark once as a test on every pull request, so none of them rots. Locally:

uv run pytest benchmarks