9.1 Fills — driving a large materialization slowly
Some materializations are too big for one query’s patience: a 12-month history over many anchors,
an open catalog sweep. A fill drives one durably, slowly, and idempotently. An operator
starts one with a query (cypher), a per-pass call budget (budgetPerTick) and a label; lists
fills with their id, state, ticks, liveCallsTotal, lastError and warning; and cancels
one by id, which keeps completed work cached and rolls nothing back. How an operator reaches
those three operations is the host’s surface; the reference host offers them under
/api/v1/admin/kg/fills.
The engine re-runs the fill’s query on a schedule, each pass under a call budget: at most
budgetPerTick producer calls, stopped CLEANLY when spent — everything fetched stands, nothing
is recorded for the skipped remainder. Because finished work is cached (and closed periods never
expire), each pass advances past everything already done and spends its whole budget on new
ground. A pass that completes with ZERO live calls is the finish line: the fill turns DONE.
Guarantees
- Idempotent and resumable by construction. The fill’s progress IS the caches. Restart the process, cancel and restart the fill, run the same query by hand in between — nothing is done twice and nothing is lost beyond the tick in flight.
- Rate-limited twice over. The per-tick budget bounds each pass; within a pass, every
producer’s own declared
cost.ratepaces its calls. - An error never finishes a fill. A failing pass records its error and stays
RUNNING— transient source trouble is retried on the next tick, and the error is visible on the fill. - A budget-exhausted ordinary query says so. Outside fills, the same budgeted stop surfaces as a warning naming what was not fetched — a partial result is announced, never passed off as complete.
A fill reaches DONE only if its producers keep what they fetch
DONE is a pass that makes no live call at all, and work already done is skipped on the next
pass only while it is still fresh. So a producer whose records expire between passes is re-read
every pass: the fill keeps working, it never runs out of live calls to make, and it never turns
DONE. Nothing is broken — it has simply become a slow permanent poller. Whether a fill can
terminate is decided entirely by its producers’ cache: declarations, and by nothing about the
query, the budget or the schedule.
That is a division of labour, not a limitation:
- A fill converges a COLD materialization of something that does not change once mined: a
closed month of history, a catalog, a backfill of records that are what they are. Declare those
producers so their records outlive the whole sweep —
cache: { kind: immutable }for reference data, or attlin days — and the fill climbs toDONEand stops.periods:already does this for you: a closed period is held effectively forever whatever the producer’s ownttlsays, which is why a 12-month history fill finishes. Only the open period rides the ordinary TTL — and it is also the reason aperiods:fill over a producer with a short TTL does not terminate either, since the open period expires under the sweep while the sweep is still running. - Re-assessing something that keeps changing — a live queue, prices, today’s state — is scheduled
work, not a fill: a Realm Function on a
schedule, or a routine bound to a signal. Those are built to re-run for ever and have no finish line to miss. A fill aimed at a live source only imitates one, at a fill’s pacing and without a schedule’s triggers.
A fill started over producers that cannot outlive a sweep is accepted and carries a warning naming them, visible on the fill in the listing above. It is a warning rather than a refusal because a slow permanent poll is sometimes exactly what someone wants; it should be chosen, not stumbled into.
A realm ships the sweep its own producer needs
The calls above are the operator’s door. A realm that declares an expensive producer should not have
to hope somebody finds out that fills exist, works out the right query and POSTs it: it declares the
sweep as a sweeps: entry on one of its own agents (README, agents/),
each naming a view of the realm’s that takes no arguments. The host then runs it as a fill with
everything above true of it — budgeted, paced, resumable, cancellable, warned about where it cannot
converge.
What a realm gets by declaring one, and what it does not:
- Installing the realm starts nothing. A realm agent arrives off duty with no sponsor and no signature, so there is no sweep to run. The declaration is visible on the agent’s card — the sweep, the view it fills, the budget it asks for — and nothing is bought until somebody sponsors the agent, signs it and puts it on duty. Those are the same three acts every realm agent already needs; a sweep adds no consent step of its own, and the install is not one of them.
- It spends the AGENT’S budget and reaches only what the AGENT may reach — the
budgetandauthority.reachits sponsor signed, never the installing operator’s own allowance. An agent whose reach does not name the API behind a producer the sweep plans calls nothing: the pass is refused before it begins. - Anybody who can stop the agent stops the sweep. Standing the agent down, retiring it or throwing the world’s kill switch stops it; so does removing the realm, which cancels the fill rather than leaving one retrying a query whose realm is gone. Completed work stays cached, so putting the agent back on duty resumes rather than restarts.
- A realm update cannot change what an adopted sweep asks. The view’s text is part of what the sponsor signed, so rewriting it shows as an unsigned change and the signed text keeps running until somebody signs again — as a routine’s body does.
- It is not a schedule. A sweep converges a cold materialization and stops, under exactly the
rules above; re-assessing something that keeps changing is a routine on a
schedule.
name: surveyorjob: Keep the crime history mined, so a question about a street answers from the graph.sweeps: - name: crime-history # stable within the agent; the fill is found by it view: street-crime-sweep # a view of this realm's, taking no arguments budgetPerTick: 60 # optional; the paid calls one pass may make| Sweep field | Required | Meaning |
|---|---|---|
name |
yes | Stable id within the agent. Asking for the same sweep twice is one fill, not two. |
view |
yes | A saved view this world has, declaring no params. Its body is the query each pass runs. |
budgetPerTick |
no | Producer calls, or inline-judgement groups, one pass may buy. Omitted takes the host’s default. |
A sweep whose view the world does not have, or whose view takes arguments, is a need on the agent:
the agent cannot be put on duty until it is fixed, so the declaration is reported before anything runs
rather than failing a pass at a time. A host may cap budgetPerTick, and the number of fills it
advances per tick is the host’s, so several sweeps in one world each advance more slowly.

