Coming from Hamilton¶

Good news: fnloom uses Apache Hamilton (incubating)'s model, so a module of Hamilton functions is already a module of fnloom nodes. Your functions don't change. Your driver code changes in a few small, mechanical ways.

fnloom isn't a drop-in replacement, though. There's no hamilton module to import and no compatibility layer. Hamilton 1.90.0 is fnloom's test oracle: every place the two differ on purpose is a row in the register of differences, and most rows have a test that checks both sides.

What you get by switching¶

• Deep graphs just work. Hamilton's graph walk recurses, so a chain of about 1,000 nodes raises RecursionError. fnloom has no depth limit.
• Faster repeat runs. prepare compiles a request once, and each run after that is a loop in Rust: up to 24× faster than Hamilton's execute on the plain graphs of the benchmarks, and more with @extract_fields.
• Typos are errors. A misspelled input or override raises instead of being silently ignored.
• Stable order. Nodes run in the same order however you list outputs, so logs and side effects are reproducible.
• Real types. fnloom ships py.typed, and its decorators keep your function's signature for type checkers.
• A lighter install. No runtime dependencies, where Hamilton pulls in numpy, pandas and more.

What carries over¶

• Name-wiring. A function's name is its node and its parameter names are its edges. A parameter that names no function is an input, and one with a default is optional.
• Modules. The public functions defined in a module are its nodes. Private and imported functions aren't.
• The builder. fnloom.Builder().with_modules(...).build() mirrors driver.Builder().with_modules(...).build(). fnloom.build(*modules) is the short form.
• execute. execute(final_vars, inputs=..., overrides=...) returns a dict of the requested outputs, in the order you asked for them.

Decorators¶

Most decorators fnloom has so far take the same arguments as Hamilton's, so usually only the import changes. @cache is the exception, below.

What changes in your code¶

What changes in behaviour¶

Caching¶

fnloom's cache keeps Hamilton's names for what they mean, and differs wherever Hamilton's cache can return a wrong value (Q46 to Q51 in the register). The guide is Caching.

Hamilton features fnloom leaves out¶

fnloom doesn't build these Hamilton features, because plain Python, or something fnloom already has, gets the same result with less to learn. Each section shows the fnloom way. Each decision is a register row.

@subdag and @parameterized_subdag¶

A namespace is a module used again (Q74), and bind= feeds each copy its own input, as @subdag's inputs= does (Q112). The function body is an ordinary function reading the namespace's nodes by their full names:

# file: taxes.py
def net(gross: float) -> float:
    return gross * 0.8


def doubled(net: float) -> float:
    return net * 2

# file: finals.py
import fnloom


@fnloom.inject(doubled=fnloom.source("eu.doubled"))
def eu(doubled: float) -> float:
    return doubled


@fnloom.inject(doubled=fnloom.source("us.doubled"))
def us(doubled: float) -> float:
    return doubled

import fnloom
import finals
import taxes

driver = (
    fnloom.Builder()
    .with_modules(finals)
    .with_modules(taxes, namespace="eu", bind={"gross": fnloom.source("eu_gross")})
    .with_modules(taxes, namespace="us", bind={"gross": fnloom.value(5.0)})
    .build()
)
driver.execute(["eu", "us"], inputs={"eu_gross": 10.0})
# -> {'eu': 16.0, 'us': 8.0}

Two things differ from Hamilton, on purpose. A misspelled parameter in a final function is a missing-input error, where Hamilton makes it a namespaced input. A bind= key no node reads is a ValueError, where Hamilton drops an unused source().

config= per copy isn't built: give each copy a config key (eu_mode, us_mode) read by @config.when. @parameterized_subdag isn't either: loop over with_modules(taxes, namespace=name, bind=...), one call per copy.

@does¶

Call the function. The node is named, typed and cached as written. A static checker checks the call against the replacing function's signature, and with_output_type_checks() checks the result at run (Q113):

# file: summing.py
def _sum_all(a: int, b: int) -> int:
    return a + b


def total(x: int, y: int) -> int:
    """x plus y."""
    return _sum_all(a=x, b=y)

import fnloom
import summing

fnloom.build(summing).execute(["total"], inputs={"x": 1, "y": 2})
# -> {'total': 3}

fnloom.node(fn, name=..., bind=...) makes the node from the function itself, with no wrapper:

driver = (
    fnloom.Builder()
    .with_functions(
        fnloom.node(summing._sum_all, name="summed", bind={"a": fnloom.source("x")})
    )
    .build()
)
driver.execute(["summed"], inputs={"x": 1, "b": 2})
# -> {'summed': 3}

@schema.output¶

Hamilton's @schema.output records a tag that nothing checks. To check a frame's columns, use @check_output_custom, which fails the node with a ValidationError by default. To record them without checking, use @tag (Q114):

# file: framed.py
import fnloom


def _has_columns(value: dict) -> str | None:
    missing = sorted({"a", "b"} - set(value))
    return f"missing columns {missing}" if missing else None


@fnloom.check_output_custom(_has_columns)
@fnloom.tag(schema="a:int,b:float")
def frame(rows: int) -> dict:
    return {"a": list(range(rows))}

import fnloom
import framed
from fnloom import ValidationError

driver = fnloom.build(framed)
driver.nodes["frame"].tags["schema"]
# -> 'a:int,b:float'
driver.execute(["frame"], inputs={"rows": 2})
# raises ValidationError: frame: check _has_columns failed: missing columns ['b']

@dataloader and @datasaver¶

A loader is a function that returns the data. If you also want its metadata as a node, return both and split them with @extract_fields. @tag marks the nodes in Driver.nodes and the export (Q115):

# file: loading.py
import fnloom


@fnloom.extract_fields({"data": list, "metadata": dict})
@fnloom.tag(kind="loader")
def raw(path: str = "p.json") -> dict:
    return {"data": [1, 2], "metadata": {"source": path, "rows": 2}}


def total_loaded(data: list) -> int:
    return sum(data)

import fnloom
import loading

fnloom.build(loading).execute(["total_loaded", "metadata"])
# -> {'total_loaded': 3, 'metadata': {'source': 'p.json', 'rows': 2}}

A saver is a node that writes and returns what it wrote, or a hook (below).

@load_from, @save_to and materializers¶

Load in a node. Its annotation is checked against each reader's at build, and its value at run with with_output_type_checks() (Q87), neither of which Hamilton's loaded values get. Save from an after_run hook, which gets the run's raw outputs (Q116):

# file: stored.py
import json
from pathlib import Path


def numbers(in_path: str) -> dict:
    return json.loads(Path(in_path).read_text(encoding="utf-8"))


def total_stored(numbers: dict) -> int:
    return sum(numbers.values())

import json
import shutil
import tempfile
from pathlib import Path

import fnloom
import stored


class SaveJson(fnloom.Hook):
    """Writes the named outputs of each successful run to one JSON file."""

    def __init__(self, path: Path, names: list[str]) -> None:
        self.path = path
        self.names = names

    def after_run(self, event: fnloom.RunEnd) -> None:
        if event.ok and event.results is not None:
            saved = {name: event.results[name] for name in self.names}
            self.path.write_text(json.dumps(saved), encoding="utf-8")


folder = Path(tempfile.mkdtemp())
(folder / "in.json").write_text('{"a": 1, "b": 2}', encoding="utf-8")
driver = (
    fnloom.Builder()
    .with_modules(stored)
    .with_hooks(SaveJson(folder / "out.json", ["total_stored"]))
    .build()
)
driver.execute(["total_stored"], inputs={"in_path": str(folder / "in.json")})
# -> {'total_stored': 3}
(folder / "out.json").read_text(encoding="utf-8")
# -> '{"total_stored": 3}'
shutil.rmtree(folder)

To choose the source per build, use a @config.when variant per source; to choose it per run, give the path as an input.

StaticValidator¶

To check the graph at build, write a function over Driver.nodes and call it right after build(). It sees the names, types, tags, dependencies, functions and config that a StaticValidator sees, and can report every problem in one error (Q117):

def checked(driver: fnloom.Driver) -> fnloom.Driver:
    """The driver, if every computed node has an owner tag."""
    problems = [
        f"{name}: no owner tag"
        for name, node in driver.nodes.items()
        if not node.is_input and "owner" not in node.tags
    ]
    if problems:
        raise ValueError("; ".join(sorted(problems)))
    return driver


checked(fnloom.build(summing))
# raises ValueError: total: no owner tag

Not there yet¶

fnloom is pre-alpha. The core, the native path, @parameterize and its sugar, @tag, @config, @extract_fields, @extract_columns, @exclude, graph inspection, validate, graphs from plain functions, namespaces, type checks, result caching, the async driver, hooks, result builders, output checks, custom value checks, drawing, export, the CLI, @check_output validators, step injection, @inject, @resolve and node wrappers are done. These Hamilton features come next, roughly in this order:

1. Dynamic execution, which has its own design.
2. GracefulErrorAdapter. Node wrappers can change how a node is called, but a run still stops at the first failure.

If your pipeline depends on one of these, it's worth waiting for it before you switch.