Skip to content
Embabel Worlds

Joining types on demand (virtual joins, not mirrored)

Population (above) eagerly mirrors a whole external collection into the graph on a schedule. For large or volatile collections you usually only ever touch a tiny slice — there a virtual join is better: the type’s instances are fetched on demand when a Cypher query traverses to them, materialized transiently for that query, then rolled back (no persistence, no sync, no GC). It’s the traversal-triggered sibling of population:.

Virtual Cypher — the engine. The host mechanism that powers on-demand joins is called Virtual Cypher. A realm never invokes it directly; you declare the pieces (virtualJoins: + producers/, and bridge resolve: chains) and it plans and runs the fetch. For a user query that traverses to a virtual label it:

  1. probes the bound real anchors the query selects — applying the query’s own WHERE / pinned-literal predicates so only the anchors that will survive are chosen (a filtered … WHERE p.name CONTAINS 'governor' resolves just those people, not the whole address book), preferring an existing real node and only seeding a transient one when none exists;
  2. plans each fetch with a cost-based optimizer — pushing predicates to the source (below), fetching per-key or batched per the producer’s declared capability (batchSafe), and budgeting calls against the source’s shared rate bucket (cost:), emitting an EXPLAIN with rewrite advice when a query can’t fit the budget;
  3. fetches the external records through the named producer;
  4. materializes them — and any brings sub-graph — as transient nodes carrying the extra :Virtual label, a dateRetrieved timestamp, and the host-bound worldId, contextId, and access-policy revision (with the acting principal retained separately for audit);
  5. runs the user’s (scope-rewritten) query over the combined real + virtual graph;
  6. rolls back — nothing persists.

Identity bridges (writeThrough, below) are the one exception: they persist as a warm cache and re-resolve after refreshAfter. The contract you write — declarative joins + producers — is the same whether the source is one record or a million; the engine handles probing, planning, fan-out caps and rollback. Execution model + worked examples (including vector/semantic edges): VIRTUAL_CYPHER.md.

A virtual type declares one or more virtualJoins:. Each says how the type is reached — from an anchor label along a relationship, keyed by an anchor field, fetched by a named producer:

# types/hubspot.yml — fetched on demand, NOT mirrored
- name: HubSpotContact
  visibility: internal
  properties:
    id: { metadata: { identity: "true" } }   # MERGE key for dedup
    email: "Primary email."
    jobtitle: "Job title."
  virtualJoins:
    # Linking on an id match: anchor is a domain node, joined by a shared property.
    - anchorLabel: Person
      relationship: HAS_HUBSPOT_CONTACT
      keyField: email            # anchor property whose values become the producer keys
      recordKeyField: email      # field on each fetched record that maps it back to the anchor
      producer: contactsByEmail  # see producers/ below
      # brings: a fetched record may carry its own sub-graph (extracted from the record)
      brings:
        - childType: HubSpotComment
          relationship: HAS_COMMENT
          records: "$.comments[*]"   # JSONPath within the record to the child list
          id: id

virtualJoins fields: anchorLabel, relationship, keyField, recordKeyField (defaults to keyField — a same-property id-match), producer, optional materializedKeyField (the property on every fetched TARGET record that holds this join’s key — see Finding a virtual node by its key below), optional brings (declared sub-graph), maxAnchors/maxFanoutTotal (caps). For a join to an external-identity node (a bridge like GitHubIdentity / HubSpotOwner), declare a resolve: rule chain + writeThrough/refreshAfter instead — see Identity bridges below (persist: true is the older eager-only form, and is refused on a join that is not a bridge — VIRTUAL_CYPHER §5.2.1). There is no flag that keeps a MINTED anchor: an unbound back-hop mints one per materializedKeyField value for the length of the query, and keeping a discovered identity is a bridge’s job. A list with more than one entry, or any join that can fan in, requires an identity property so convergent paths dedupe to one node.

The query may only reach a virtual label by traversing a declared join from a bound anchor — a naked MATCH (hc:HubSpotContact) is rejected. Every materialized node carries the extra :Virtual label, a dateRetrieved ISO-8601 timestamp, and the host-bound worldId, contextId, and access-policy revision (so the normal scope rewriter matches it) — bookkeeping that labels(), keys() and properties() leave out, so a fetched node reads as the stored record would (VIRTUAL_CYPHER.md §2); the user’s query runs through the rewriter unchanged, and the whole materialization is rolled back when the query completes.

A query may also reach a virtual node by pinning the anchor with a literal — MATCH (g:GitHubIdentity {login:'octocat'})-[:RAISED]->(i:GitHubIssue) — even when no GitHubIdentity{login:'octocat'} exists in the graph. The literal (inline {...} or a WHERE alias.login = '…') seeds a transient anchor, so a producer can be keyed on a named identity (any GitHub login, not just the connecting user’s), fetched with the connecting user’s credentials. Multiple joins onto the same virtual node compose: (me)-[:RAISED]->(i)<-[:ASSIGNED]-(:GitHubIdentity {login:'octocat'}) materializes both sides and intersects them.

Finding a virtual node by its key. A join may declare materializedKeyField — the property on every fetched record of the TARGET that holds the join’s key:

- name: BiblePerson
  virtualJoins:
    - anchorLabel: BibleNameQuery     # a lookup anchor: identity `name`
      relationship: NAMED
      keyField: name
      recordKeyField: nameQueried
      materializedKeyField: name      # every BiblePerson record carries the looked-up name in `name`
      producer: biblePeopleByName

Then the target can be matched by that property directly, with no door written: MATCH (p:BiblePerson {name:'Moses'}) — or WHERE p.name = 'Moses', WHERE p.name IN [...], or a value bound upstream — is answered as if (:BibleNameQuery {name:'Moses'})-[:NAMED]->(p) had been written, and p anchors any join that leaves it ((p)-[:FATHER*]->(a:BiblePerson)). The query’s own predicate still applies to the fetched rows with Cypher’s semantics, so a producer that matches keys case-insensitively may fetch rows an exact = then filters out. The declaration is required: a target property that merely shares the key’s name is never treated as the key, because fetching by a coincidence of spelling answers a different question with full confidence. Without it the bare match is still rejected as a naked scan, and the rejection names the keyed form when one is declared. When several joins declare a materializedKeyField the query pins, the first in declaration order is taken.