Producers are the source-specific fetchers a virtualJoins.producer references — declared once, reused. Conceptually each is a Repository over an external store (the Spring Data analogue: one Repository abstraction, different stores underneath); the kind discriminator picks the store. Each .yml in producers/ is a list. Batch contract: a producer takes ALL anchor keys at once and returns the matching records — never one call per key (no N+1).
producers/hubspot.yml
- name: contactsByEmail
kind: remote# a RemoteRepository — gateway op (realm handler or learned API)
operation: objectsSearch
queryArgs: { segment: string } # typed request input from an edge's realm map
sourceFilters: [segment] # source capability; does not itself narrow a request
records: "$.results[*].properties"
keyArg: "filterGroups.0.filters.0.values"# where the key LIST is injected (list mode)
datasource: warehouse# a realm/world SQL datasource (sql/datasources.yml)
query: "SELECT id, customer_email, total FROM orders WHERE customer_email IN (:keys)"
For kind: remote, queryArgs declares scalar request inputs (string, integer,
number, boolean), supplied as realm values on the edge. They are optional unless named in
requiredQueryArgs, which must list only declared queryArgs; a read missing a required input is
refused, never sent bare. Undeclared, ambiguous or invalid inputs are rejected, not silently dropped. sourceFilters lists filter-capable source fields;
listing one does not send it. An unsent condition on a declared filter can yield
UNPUSHED_FILTER when a bounded read cannot establish a match (see Virtual Cypher §7.2.2).
Producer kinds:
kind
fetch
notes
remote (alias api)
a gateway.<name>.* op (realm handler or learned API) — a RemoteRepository
list mode (keyArg → array) or string mode (keyTemplate + {keys}); records JSONPaths the response
sql
a SELECT against a host-configured realm/world datasource, or a read-only stored procedure’s result set (procedure:)
keys expand into IN (…); rows are the records; SELECT-only. The host mediates approved credentials and operations; governed captured callers require a retained resource receiver. A procedure: is called once per key on a read-only datasource and rolled back after the read — see Virtual Cypher §5.15.1
compute
an in-process computation over the keys
scores / rollups / synthesis — no external I/O; local, so NOT a RemoteRepository
vector
top-k semantic similarity to the anchor
for joins with no key — similarity is the join (related docs/chunks); rides the host embedder. index: memory is what the assistant remembers about the acting user, under the shared Memory label — see Virtual Cypher §6.8
generative
GENERATES the edge (resumably) rather than reading it — an LLM’s world knowledge (SIMILAR_TO, IN_INDUSTRY) or a code function
pluggable generator (llm | function); keeps generating; resolves each answer onto the type spine; provenance-stamped
aggregate
REDUCES an anchor’s connected neighborhood to ONE node (fan-IN) — e.g. a per-principal or per-organization summary distilled from many rows
gathers via a scoped graph read; delegates the reduction to an existing LLM aggregation (synthesize/summarize/…); TTL-cache = periodic refresh
jar
a Java jar’s classes, read from the jar when a query reaches it — types, fields, methods, field accesses, calls
emit: the collection, by: the row property the key selects on; no ingestion, nothing stored — see Virtual Cypher §5.18
source-tree with from: jar
the source a jar was compiled from — files, spans, declarations, call sites, locals
emit:, roots: (folders, -sources.jars, {git, ref}); keyed by a jar row’s sourceKey / spanKey — see Virtual Cypher §5.19
Naming:kind: remote is the current spelling for an externally-backed repository; kind: api is accepted as a back-compat alias and still works in existing realms.
Where remote/sql/vectorretrieve records from a store and compute derives them once, a
generative producer generates them — and can be asked for MORE (the generator model). The
generation is pluggable via generator.kind:
llm — the model’s world knowledge. The prompt is authored by the realm, inline; the host’s
generative backend is domain-agnostic and ships no prompt of its own. Parametric, so a volatile fact is
refused (it would be confidently stale).
function — a host/realm-supplied resumable function (a GeneratorFunction bean), the
Python-generator analogue: given the keys, the exclusion set, constraints, demand and round, it yields
candidates. No LLM, no volatility gate.
The rest — resolving each answer onto the type spine, provenance, and demand-driven re-probing — is shared.
- name: similarMovies
kind: generative
edgeType: SIMILAR_TO# the edge this fills (host may whitelist which edges generate)
identityField: imdbId# the target type's identity — records carry it after resolveVia
anchorKeyField: similarTo# record field the anchor key is echoed into (links each answer to its anchor)
nameField: title# the human name the generator emits (dedup/exclusion happen in THIS space)
volatility: static# static | slow | volatile — a volatile fact is refused for an `llm` generator
confidenceFloor: 0.3# drop answers below this confidence
defaultWant: 25# a view has no LIMIT; keep generating until this many SURVIVE the filters
For each numbered item, name similar ones a fan would enjoy…
{% for a in anchors %}{{ a.n }}. {{ a.title }}
{% endfor %}
{% if constraints %}Only suggest items where: {% for c in constraints %}{{ c }}; {% endfor %}{% endif %}
resolveVia: # optional: resolve each emitted name onto the type spine (a nested remote op)
kind: remote
operation: getMovie
args: { t: "{keys}" }
project: { imdbId: imdbID, title: Title, year: Year, genre: Genre } # fill the type's FULL property surface
A function generator instead:
generator:
kind: function
operation: mySimilarFn# a GeneratorFunction bean: yield(keys, exclude, constraints, want, round)
Semantics (both generators):
Resumable / demand-driven. The host re-probes (pushing a growing exclusion set into the generator)
until want records survive the query’s filters — a query LIMIT, else defaultWant. So a
heavily-filtered view (most rows knocked out by a genre or availability filter) still fills up.
Constraints pushdown. Target-node predicates in the query (e.g. WHERE m.genre CONTAINS 'Noir') reach
the generator as constraints, so it only proposes matching answers, and they count toward survival.
Rejection feedback. Constraints alone are not enough when the filtered property is one the generator
cannot know precisely from memory — an exact runtime, a release year, a page count. So after each round the
host feeds the REJECTED candidates back to the generator with the real values they were rejected on
(The Kid (runtimeMinutes=68)), taken from the resolved record. The generator re-aims from evidence
instead of guessing again, and a narrow range (runtimeMinutes >= 80 AND <= 90) stops starving. This is
automatic for an llm generator: the host appends the correction itself, so a realm prompt needs no new
variable and cannot forget to opt in. A function generator gets the typed constraints and evaluates
them itself, so it is not sent the feedback. Coverage only — the query’s WHERE is still the filter of record.
Starvation is reported, never silent. If a round produces real candidates and the query’s filters reject
every one, the result carries a FILTER_STARVED warning naming the steer, the filters, and a sample of the
near-misses with their values — enough for the answer to say “18 films matched your taste, but the shortest
ran 94 minutes” rather than a bare “no results”. Read the warnings, always.
Fill the whole type. A producer materialising a typed node fills that type’s FULL property surface
(via resolveVia.project), not just its identity.
Provenance. Each record is stamped _source (the generator kind — llm/function), _confidence,
_asOf.
cache: is orthogonal to kind (none / ttl / session / immutable).
Every other producer is fan-OUT: one anchor → many target records. An aggregate producer is the
fan-IN mirror: it gathers the anchor’s connected neighborhood and reduces it to ONE node. Use it when
the reduction should itself be a node you can traverse to and cache — a per-principal summary, a per-org rollup, a
per-topic digest — rather than a scalar computed inline.
It does not reimplement the reduction: it delegates to an existing LLM aggregation (synthesize /
summarize / themes / … — the same functions a query can call inline as synthesize(text, goal)). So there
is one implementation of “LLM-reduce a group of text”, whether you write it in Cypher or declare it on a
producer. The neighborhood traversal, the per-item text, and the reduction goal are all realm-authored — the
host ships no domain prompt and no model (the aggregation uses the ops-controlled aggregation LLM).
# producers/movie.yml — one MovieTasteSummary node distilled from all of a user's ratings
- name: movieTasteSummary
kind: aggregate
edgeType: HAS_MOVIE_TASTE_SUMMARY
identityField: anchorKey# ONE node per bound anchor inside the host-bound world
anchorKeyField: anchorKey# the join's recordKeyField — links the one node back to the anchor
collect: # the fan-IN neighborhood: a scoped read (a:anchorLabel)-[:via]->(t:targetLabel)
anchorLabel: AssistantUser
via: RATED
targetLabel: MovieRating
text: "{{ title }} — rated {{ rating }}/10"# Jinja per neighbor node → one text item for the reducer
where: "t.rating >= 1"# optional extra predicate on the neighbor alias `t`
reduce:
using: synthesize# a registered LLM aggregation
into: summary# the record field the reduced value lands in (a declared property on the type)
args: # aggregation args — e.g. the GOAL for synthesize
- "In ~100 words, second person, sum up this person's taste in film."
The node is virtual like any other: reached only from a bound anchor ((me:AssistantUser)- [:HAS_MOVIE_TASTE_SUMMARY]->(ts:MovieTasteSummary)), materialized on demand, rolled back after the query. Give
its type a Movie/Foo prefix so the label and edge can’t collide with another realm’s summary type.
A join is declared under the type it PRODUCES — and chains (multi-stage)
Rule (easy to get wrong): a virtualJoins: entry is declared under the target type — the type it
materializes — with anchorLabel naming where it starts. SIMILAR_TO produces Movie, so it lives under
Movie with anchorLabel: MovieRating; AVAILABLE_ON produces StreamingService, so it lives under
StreamingService with anchorLabel: Movie. A join does not go under its anchor type. Put it under the
wrong type and the planner registers it against the wrong target and it silently never fires.
The anchor of one join can be the virtual target of another, and the engine stages them in one read tx:
Keep every hop pattern-bound. The probe follows MATCH patterns. A node re-bound out of a list — WITH collect(run) AS runs … UNWIND runs AS f MATCH (f)-[:HAS_JOB]->(j) — is NOT probed for its joins: the second hop fetches nothing and the query returns no jobs, with no warning (verified 2026-09-23). Narrow with WITH run ORDER BY … LIMIT n and hop from run itself. OPTIONAL MATCH over a virtual hop is likewise not probed; use MATCH.
StagedVirtualCypher materializes stage 1, treats its target as real, re-probes, then materializes stage 2 off
it (up to MAX_STAGES deep). So a fan-IN → fan-OUT pipeline is expressible — reduce a user’s ratings to one
MovieTasteSummary node, then generate films from that summary. Both joins go under the type each produces:
# types/movies.yml — BOTH joins under Movie (what they produce), not under their anchors
- name: Movie
virtualJoins:
- anchorLabel: MovieRating# films similar to ONE rated film (fan-OUT)
relationship: SIMILAR_TO
keyField: title
producer: similarMovies
- anchorLabel: MovieTasteSummary# films matching the WHOLE taste (fan-OUT off the fan-IN summary node)
relationship: SUGGESTS
keyField: summary# the MovieTasteSummary.summary prose is the generator input
recordKeyField: fromTaste
producer: tasteBasedPicks# a `generative` producer; MovieTasteSummary is an `aggregate` node
The intermediate node is transient (materialized then rolled back with the read) — it need not be persisted
for the downstream join to see it, because both stages run in the same read tx. Give the summary type a TTL
cache: (weekly) so the expensive fan-IN reduction is reused across queries within the window.
Predicate pushdown (pushdown:) — scope the fetch at the source
By default the graph filters after materialization: a virtual join fetches broadly, then the query’s WHERE drops non-matches. For a prolific anchor that’s wasteful and can hit the source’s result cap before the matches you want. Pushdown translates a query predicate on the virtual target node into the source’s native filter so the fetch is scoped before it returns — the Spring Data analogue is pushing a Specification/Criteria to the store, with whatever can’t be pushed still filtered in the graph (so correctness never depends on pushdown, only cost and coverage).
A remote repository declares pushdown: rules; the host renders matching predicates into the {filters} placeholder of args:
pushdown:
- property: html_url# the target-node property the predicate is on
op: contains# EQUALS (default) or CONTAINS
qualifier: "repo:{value}"# native fragment; {value} ← the predicate's value
valuePattern: '…([\w.-]+/[\w.-]+?)(?:/|$)'# optional regex; group 1 replaces {value}
So WHERE i.html_url CONTAINS 'acme-corp/widgets' turns is:issue author:X {filters} into is:issue author:X repo:acme-corp/widgets — one scoped search instead of fetching the author’s thousands and intersecting in the graph. The mapping is declarative and source-specific; the engine knows nothing of repo:.
Only a LITERAL is pushed.run.created_at >= '2026-09-16T00:00:00Z' renders into the source’s filter; run.created_at >= toString(datetime() - duration({days: 7})) does not — the value is not known when the fetch is planned, so the fetch is broad and the comparison is applied in the graph. A view’s declared params become literals before the query is read (VIRTUAL_CYPHER §7.6.1), which is how a windowed view gets a pushed-down window: take since as a parameter and let the app or the asker supply the date. A relative window (“the last 7 days”) cannot be pushed by any spelling; say so in the view’s description and bound the fetch with paging.maxPages.
Three consequences of “attached” predicates, verified the hard way (2026-09-23) and the first two re-measured (2026-10-17) now that a predicate over a projected name is pushed like any other:
Attachment decides what pushdown: sees; what the fetch ASKED its source for decides the cache key. Every predicate on the target node is attached (the shapes in VIRTUAL_CYPHER §7.6.1), and attachment does not stop at a WITH: MATCH …->(run) WITH run WHERE run.status = 'completed' is attached exactly as if it were in the MATCH’s own WHERE, so is a wrapped one (toInteger(run.id) = 123), and so — since the engine began reading a predicate through the name a WITH introduced — is one written over a projected name (WITH run, run.created_at AS createdAt WHERE createdAt >= $since), which is the predicate on the property it names and is pushed exactly as that property’s is. The KEY is the narrower set: the filter a rule absorbed and rendered into the request, plus a LIMIT the planner could push. A predicate no rule absorbs — a fold, a property with no rule — is attached, is seen, is not sent, and is no part of the key, so the graph applies it to cached records exactly as to fresh ones. Two views whose REQUESTS differ are two keys; a cached complete read serves the narrower of the two (the unfiltered read serves the filtered view, a pushed LIMIT 50 serves LIMIT 10 in the same declared order) and never the broader, so the order the views run in decides the calls (VIRTUAL_CYPHER §5.13). No spelling makes a family share one read. The trick this bullet used to teach — filter on projected variables, because such a predicate attached to nothing — worked only while that was a defect, and the engine has since fixed it: a predicate over a projected name attaches and is pushed like any other. What works instead, measured on a family of three windows over one door (2026-10-17): let the broad view read FIRST and every window after it is free in any order (a window read first serves neither the broad question nor a wider window, and each pays its own call); limit a terminal result by list slice (ORDER BY … WITH collect(row) AS rows UNWIND rows[0..$limit] AS row) rather than by a LIMIT the planner can push, which leaves a prefix and not the collection; keep ONE WITH … LIMIT directly before a hop, carrying its order key (WITH run, createdAt ORDER BY createdAt DESC LIMIT $n MATCH (run)-[:HAS_JOB]->(j)), where it bounds the hop’s anchors (a LIMIT that feeds a later stage sets no demand target, and a WITH between it and the hop hides it); to bound a hop to ONE record, rank and take one (WITH run, CASE WHEN toInteger(run.id) = $id THEN 0 ELSE 1 END AS pick ORDER BY pick LIMIT 1 MATCH (run)-[:HAS_JOB]->(j) WITH j, pick WHERE pick = 0); and where a family must share one read WHATEVER the order, read it through a door that absorbs nothing — a periods: door, whose window is a key rather than a narrowing, or a door declared readWhole: true, which keeps its pushdown: rules and sends none of them, so every view of it shares one read whatever the order (give it a ttl or immutable cache, or there is nothing to share). Measured on a periods: door: these shapes took a 12-view app from 91 s to a few seconds, every view 0 calls after the first read.
Two traversals of the same door in one query share ONE fetch, and what that fetch asks is decided for the group. So a predicate on one traversal can scope the records the other sees — a summary over all runs beside a second MATCH over the same door WHERE f.conclusion = 'failure' is a figure worth checking against the same figure read through a query that reads the door once. Measured 2026-10-17 on the shipping engine: a count over every run beside a filtered second MATCH of the same door asks the source for the whole collection, so the total is over everything and the filtered count is its own, in either spelling. Do not reach for a spelling to arrange that: no spelling un-attaches a predicate from its node any more (bullet 1). The levers are a sort-and-LIMIT on a WITH, and giving the second reading its own door.
A predicate that is attached is also what pushdown: sees; one that is not attached is never pushed. The two mechanisms read the same set.
Verify that a pushdown actually narrows — some sources ignore unknown filters SILENTLY. The engine cannot tell a filter the source honoured from one it discarded: both return 200 with records. A source that responds to an unrecognised filter key by returning the entire unfiltered collection turns a typo, a renamed upstream field, or an optimistic guess into a full-collection scan that looks like a success — the query still returns correct rows (the graph filters what pushdown didn’t), so nothing fails; you just quietly fetch everything, every time. This is real: the NSW planning feed used by realm-nsw-property returns all 426,096 records for a misspelled filter and never errors.
Before declaring a pushdown: rule, call the source twice — once with the filter, once without — and confirm the counts differ. Declare rules only for keys you have proven narrow, and say so in a comment. When a source’s filter surface is partly unsupported, the honest producer declares the few verified keys and leaves the rest to graph-side filtering; document which properties do not push down, because a query author will otherwise assume a WHERE on any property is cheap.
A source that applies no filters declares pushdown: []. Some endpoints take nothing: a conversation’s messages come back as the whole thread or not at all; a list keyed by status accepts no other parameter. Writing pushdown: [] is the author saying so, and it is a different statement from leaving the key out. An absent key states nothing — the engine cannot tell “I have not got to it yet” from “there is nothing to declare” — so it warns at load that the producer declares no pushdown:, notes on every read of 25 records or more that a WHERE was applied in the graph and advises a rule (FILTERED_IN_GRAPH), and ends EXPLAIN’s residual line with the same advice. A declared empty list clears all three: the load warning is not raised, the read’s cost note says the source applies no filters and advises nothing, and EXPLAIN names the residual without an instruction. What it does not change is the note itself, which stays on the read: it is the evidence that the read was complete and the graph applied the filter, which is what lets an empty answer be reported as an exact empty rather than as records that were fetched and matched nothing. There is no field to cross-check it against, because the declaration is the pushdown: key: a producer carries either rules or an empty list, never both. Declare it only when the source really takes nothing — it cannot be undone by the engine, and a rule the source would have applied is then never written.
A door meant to be shared declares readWhole: true instead. It sits beside pushdown: on any kind that declares rules, and it is not a cache setting: it changes what the source is asked. Every read asks for the key’s whole collection, the declared rules stay declared and are not sent, and the graph applies every filter. Every read is then keyed the same, so with a ttl or immutable cache a family of views over the door — windows over one listing, a summary beside its detail — costs one read in any order, where a door that sends its rules pays one read per window unless the broad view happens to run first. The read keeps its cost note, which names the declaration and advises no rule; the quote and EXPLAIN put the filters on the graph’s line. Without a ttl or immutable cache there is nothing to share and loading warns; with requiresFilter: true it is refused. Keep the rules on it, so a second, narrow door over the same operation can send them:
- name: runsWhole# the family's door: one read per repository, shared by every view
remote is not the only kind that can filter at its source. A sparql, sql, cypher, tabular or feed producer declares pushdown: too. Each entry names a property the source can filter on, the operators it applies, and the type its values take at the source:
- kind: sparql
name: saintsOfCountry
endpoint: https://query.wikidata.org/sparql
keyVar: country
query: |
SELECT ?country ?saint ?died WHERE {
VALUES ?country { {{keys}} }
?saint wdt:P27 ?country ; wdt:P570 ?died .
{{filter:died}} # the filter goes HERE, before the label lookups
type: dateTime# string | integer | decimal | boolean | date | dateTime | iri
ops: [GREATER_THAN_OR_EQUAL, LESS_THAN] # omit to allow every operator the type supports
Where the filter goes depends on the kind:
Kind
Where the filter is applied
sparql
At its {{filter:<property>}} slot, as FILTER(<expr> <op> <literal>). expr defaults to ?<property>. For an aggregate, declare having: true with the aggregate as expr (expr: "COUNT(DISTINCT ?s)", slot after GROUP BY), and the slot renders HAVING(…).
cypher
At its {{filter:<property>}} slot, as AND (<expr> <op> $param), so place it after an existing condition (WHERE o.customer IN $keys {{filter:total}}). expr is required (expr: "o.total").
sql
Appended to the generated statement as AND (<column> <op> :param). The property is the column. No expr: the statement stays generated.
tabular, feed
Applied while reading, before a row becomes a record or counts against a per-key cap. A feed filters on title, url, publishedAt or summary.
generative, aggregate, agentic-rag
Not supported. Every predicate is applied after the fetch.
The operators are EQUALS, CONTAINS (text only), GREATER_THAN, GREATER_THAN_OR_EQUAL, LESS_THAN, LESS_THAN_OR_EQUAL and IN.
Rules:
Only the author knows where the filter saves work. The host can always wrap a query and filter its result, but that saves the source nothing. The slot puts the filter inside the query, before the expensive part. If nothing is pushed, the slot renders as empty text.
The host binds or escapes every value; none is spliced in raw. Each value is parsed as the declared type and written from the parsed value: a typed RDF literal ("20"^^xsd:integer, an xsd:dateTime, an escaped string), or a bound SQL or Cypher parameter. A value that does not parse as its type, or an operator the entry does not list, is not pushed. It stays a filter the graph applies after the fetch and never causes an error.
The cache key holds exactly the predicates pushed. A predicate the producer pushes changes what the source returns, so it is keyed. A predicate it does not push shares the cached fetch with the unfiltered query, and a predicate it does push is served from a cached complete read of the same key taken without it (VIRTUAL_CYPHER §5.13).
A pushed filter must be exact. The graph still applies every predicate after the fetch, so a looser source filter only costs extra rows. A stricter one, such as an equality that misses a language-tagged literal, loses rows nothing can recover. Compare through the expression that makes the source agree with the graph, for example expr: "STR(?label)".
Validation.realm_validate reports an error for a declared entry whose slot is missing from the query, and for an entry the kind cannot honour (having without expr, a cypher entry without expr, an sql entry on a stored procedure or an unexposed column, a field a feed item does not have). It warns about a slot with no entry, and about a declared property the query does not appear to project.
project paths: use [*], never [0]. An INDEXED path is not honoured and projects null silently — no warning, no error, just an empty property, and any predicate over it then drops every row. Only the [*] form reaches into a nested array, and it yields a list, so a single-valued nested field arrives as a one-element list the consumer must unwrap. Write address: "Location[*].FullAddress", not Location[0].FullAddress. (Verified 2026-07-28: with [0], every address and coordinate in a fetched collection was null while the flat scalar fields projected fine — the kind of defect that reads as “the source didn’t return that field”.)
project takes paths, not expressions. A JSONPath filter (steps[?(@.conclusion=='failure')].name) or function (steps.length()) projects null silently, exactly like an indexed path. Project the parallel lists (step_names: "steps[*].name", step_conclusions: "steps[*].conclusion") and derive in Cypher: [i IN range(0, size(j.step_names) - 1) WHERE j.step_conclusions[i] = 'failure' | j.step_names[i]].
Relatedly, a declared type is what a value arrives as; an undeclared property arrives as the source encodes it. A JSON feed that quotes its numbers sends strings. When the target type declares the property int/long/decimal/number/boolean, the fetched value is stored as that type — "35703978620" becomes a number, so WHERE run.id = 35703978620 matches; a declared list is converted element by element. A property with no declared type keeps the source’s own value, so WHERE n.cost >= $min on a quoted cost compares a string to a number and matches nothing: declare the property’s type, or coerce at the point of use (toFloat(n.cost)). A value the declared type cannot read ("n/a" in an int property) is kept as sent rather than failing the fetch, and a numeric comparison against it does not match — say in the property’s description when a source sends such values. And beware that a null-propagating comparison also removes rows with no value at all — which for something like a cost bound means records the user would have wanted to see silently vanish.
Composite-key joins — when no single anchor property is the key. A source keyed on a pair — a latitude/longitude point for a geospatial lookup, an owner/repo slug — declares the join with producerKeyFields and omits keyField:
- anchorLabel: WeatherLocation
relationship: HAS_CURRENT
producerKeyFields: [latitude, longitude] # composed, in this order, into ONE key per anchor
recordKeyField: key# the record field carrying that key back
producer: weather.current
The engine composes the named anchor properties, in order, into one opaque key per anchor and hands those to the producer. The link is the whole composite: each record must carry the exact key it was fetched for under recordKeyField — a remote producer gets this for free by declaring echoKeyAs: key; a TypeScript handler echoes the key it received. A record never links to an anchor that merely shares one component, and an anchor with any component absent contributes no key (the same null rule as a single keyField).
A remote producer destructures the composite back into separate request arguments with keyArgs, naming one argument per component in the same order:
keyArgs: [latitude, longitude] # → latitude=…&longitude=… per call
echoKeyAs: key
keyArgs implies one source call per key. The composite’s own separator is reserved by the engine and always takes precedence over keySplit — and the two do not compose: a composite component is handed to keyArgsas one argument, never re-split by keySplit. So a source-shaped component like "owner/repo" does NOT become two path parameters; declare the parts as separate anchor properties (producerKeyFields: [owner, repo, id], projected from the record that produced the anchor) and name one keyArgs entry per part. (Verified 2026-09-23: producerKeyFields: [repository, id] with keyArgs: [owner, repo, run_id] sent owner=embabel/embabel-agent, repo=<the id> and no run_id, a 404 that read like a missing run.)
A composite that is itself echoed (echoKeyAs: runKey on a first hop) may be a component of the NEXT hop’s composite: producerKeyFields: [runKey, id] splits back into every part of runKey followed by id, so a three-hop chain (repository → run → job → annotation) threads the path parameters down without any record having to carry them. Name every part in keyArgs, in order, even the ones the operation does not declare — an undeclared name is simply not sent, and it keeps the positions aligned.
A key with an empty path segment is refused, not sent. A URL (https://…, whose // is an empty segment) is therefore never a usable key for a path-parameter producer, and keySplit: "/" over one is refused with a warning naming the key. Compose the parts instead, as above.
project itself still maps flat paths only — there is no template or concatenation form insideproject; composition is the join declaration’s job, as above. A TypeScript handler (src/api/*.ts) remains the escape hatch when the source needs more than destructured arguments.
A search/list op returns one page; paging: makes the producer walk pages and accumulate, bounded by maxPages, so a scoped fetch that still exceeds a page is fully captured (and a cross-join intersection doesn’t silently miss matches past page 1).
page (increment param from startPage) or cursor (opaque).
param / sizeParam
page / per_page
Page-number arg, and the page-size arg.
startPage
1
Page-number style only: first non-negative page number sent to the source. Set 0 for zero-based APIs.
size
100
Records per page (set to the endpoint’s max).
maxPages
5
Hard cap on pages fetched, independent of page numbers. startPage: 0, maxPages: 2 fetches pages 0 and 1.
cursorParam / cursorPath
after / —
Cursor style only: request arg + JSONPath to the next cursor. startPage is ignored.
Omitting startPage preserves one-based requests (1, 2, …). A negative value is invalid. For either starting convention, a short page ends the walk normally; a full final page at maxPages reports the existing truncation warning.
A listing that may be one page: singlePage: true. A remote producer without paging whose
operation declares a page, offset or cursor argument but no page-size argument may answer only the
page the source chooses, often its default first page. Declare singlePage: true on such a producer
(a learned OpenAPI listing does this automatically). Every non-empty read then reports
PARTIAL_RESULT (TRUNCATED), and nothing counted or summed through it is presented as an exact
total. An empty read is still a complete answer. The warning also appears when the source did return
everything, because the producer cannot tell. Raising maxPages or asking again cannot clear it;
declaring paging: so the listing can be walked does.
An operation that declares no paging and no page size returns its whole collection as far as a
client can tell, so its reads are complete. A learned listing that declares a page size sends it at
the declared default, and only a full page reports PARTIAL_RESULT (TRUNCATED).
param / sizeParam name parameters declared on the operation, so an API that takes paging somewhere other than the query string is addressed by naming the parameters it actually declares. A handful of APIs pass paging (and even filtering) as HTTP headers — the NSW planning feed used by realm-nsw-property takes PageSize, PageNumber and a JSON filters string as headers. Declare them as in: header parameters in the vendored spec and name them here; the walker then drives them correctly (verified against a live world, 2026-07-28).
Always set param/sizeParam when the endpoint’s paging arguments are not literally named page and per_page. The defaults are injected as query parameters, and a source that ignores unknown query parameters — many do, silently — will return page 1 for every request. The walker cannot detect this: it sees maxPages successful 200s with a full page of records each time, and reports the product as the record count. In the case that motivated this note it reported 3,200 records that were 400 records repeated eight times, with no warning, and every downstream statistic was computed over eight copies of the same page.
The failure is silent by construction, so verify rather than assume: check that page 2’s records differ from page 1’s before trusting a paged producer. A count that is suspiciously close to size × maxPages is the tell.
A failed fetch produces zero rows — the warnings are what tell you it failed. NEVER discard them. When a producer errors (a timeout, a 401, a cancelled request) it contributes no records, and the rows are then indistinguishable from a legitimately empty result. The engine does surface the failure: gateway.kg.query returns an unconditional {rows, warnings} envelope, and a fetch failure lands there as a diagnostic. A consumer that reads result.rows and ignores result.warnings throws away the only signal separating “the source says there is nothing here” from “we never found out”, and renders a broken fetch as a confident negative.
// A zero count with a FETCH_FAILURE warning is NOT a finding — report it as incomplete.
For most joins, silently degrading to zero is tolerable. For any question where absence is the reassuring answer — is anything being built near this address, are there recalls on this product, does this person have open issues — it is the dangerous direction: a zero must never be presented as a finding when a warning says the fetch did not complete. Withhold the derived figure rather than computing a rate over a failed fetch.
This bites hardest on a paged walk of a slow or throttling source: latency climbs page over page until the request is cancelled part-way, so the fetch that fails is the one that would have returned the most data — the large result set, not the empty one.
Chunking (maxKeysPerCall). Producers chunk the unioned anchor keys into batches of maxKeysPerCall — so a traversal over many anchors stays within the endpoint’s limit and never becomes N+1. Set it to the endpoint’s documented cap: a search IN/OR (query-length bound, ~50, the api default), a dedicated bulk-by-ids endpoint (HubSpot /batch/read 100, Jira bulkfetch), or a $batch/composite multiplex (Microsoft Graph 20, Salesforce 25). sql defaults to 500.
Per-key vs batched (batchSafe). A producer batches up to maxKeysPerCall keys per call by default. Set batchSafe: false when one call covering many keys is not complete per key — a globally-ranked, capped search is the classic case: GitHub issue search author:a author:b returns ONE updated-desc list capped at paging.maxPages × size, so a prolific author fills the cap and a low-volume colleague’s results fall off the end (you’d list them for one question and find nothing for the next). With batchSafe: false Virtual Cypher fetches one key per call, giving each key its own budget. It is a declared capability, not a magic number — you do NOT also shrink maxKeysPerCall, so a realm can’t reintroduce the starvation bug by forgetting to. (echoKeyAs already implies per-key.)
There is a second, more mundane reason to set it, distinct from the starvation case above: the source is structurally single-key — its endpoint accepts exactly one value and there is no batch form. A point-in-polygon lookup (geometry=lng,lat), a “get by id” path parameter, and a single-document fetch are all of this shape. Batching is not merely incomplete there, it is impossible, so declare batchSafe: false and size cost: for the fan-out you will actually incur. When the producer also stamps the key with echoKeyAs, per-key is already implied and batchSafe is redundant — but stating it costs nothing and documents the source’s shape for the next author.
Conversely, do not reach for batchSafe: false just because a source looks single-valued: check whether the filter accepts a list. A source that takes one name per call in its examples may accept several (the NSW planning feed’s CouncilName takes an array, verified by comparing counts for one council against two), and a genuine batch is an order of magnitude cheaper than a per-key fan-out.
Search vs bulk-by-ids. When the join key is the target’s own id, prefer the source’s dedicated bulk-read endpoint (operation: = that op, keyArg: = its ids array, maxKeysPerCall: = its cap) — it’s cheaper than a search and supports field/expand selection. Use search (IN/OR) only when the key is a secondary field (email, login, domain), where there is no by-id endpoint. Pair brings with the endpoint’s expand/include/sideload params so the sub-graph arrives in the same call rather than a follow-up. If a search result doesn’t carry the key you searched by (so recordKeyField has nothing to match), set echoKeyAs: — the producer then calls one key per call and stamps the queried key onto each record (see Identity bridges).
Field selection & cursor paging (set them in args/paging). Two efficiency levers that need no producer code — just declare them:
Field masks / sparse fieldsets — request only the fields the join needs via the op’s field param (fields, $fields, X-Goog-FieldMask, properties: [...]). Smaller payloads, lower latency, less server CPU. Across Google, Zendesk, Jira, HubSpot this is the single cheapest win.
Cursor paging — prefer style: cursor over offset in paging:. Beyond performance, some sources (e.g. Slack) grant a higher rate-limit tier to cursor-paginated calls than un-paginated ones, so cursor paging is a quota win too.
These hold across the ~13 SaaS APIs surveyed (Jira, Salesforce, Microsoft Graph, Shopify, HubSpot, Stripe, GitHub, Zendesk, Google, Slack, Notion, Airtable). The very tightest (Notion 3 req/s with no bulk endpoint; Airtable 5 req/s) make cache: and chunking mandatory, and are the case for the (deferred) $batch/composite multiplex producer kind.
