lane: documents — a collection that lands as searchable DOCUMENTS

Everything above mirrors records into the graph. A collection of long-form, reasonably stable content — a published text, a documentation site, a regulator’s guidance, a policy library — belongs in the document store instead: chunked, embedded, and reachable by document search and relevance joins (RELEVANT_TO). lane: documents declares that, with no handler code: the host reads the records, renders each into a document, and keeps them current.

sources.yml

sources:
  - name: bible-kjv
    lane: documents                      # graph (default) | documents
    from:                                # exactly one of:
      cypher: |                          #   a read over data the world holds (typically this realm's reference/)
        MATCH (b:Book)<-[:IN]-(p:Passage)<-[:IN]-(v:Verse)
        WITH b, p, v ORDER BY v.verse
        RETURN p.osis AS osis, p.name AS title, collect({n: v.verse, text: v.text}) AS verses
      # urls: [https://…, https://…]     #   pages the host fetches
    document:                            # Jinja; each record's columns are the model
      uri: "bible://kjv/{{ osis }}"      # the replace key — a stable `<scheme>://<id>`
      title: "{{ title }} (KJV)"
      content: |                         # markdown preferred: headings drive chunking
        # {{ title }}
        {% for v in verses %}{{ v.n }} {{ v.text }}
        {% endfor %}
      # url: "{{ link }}"                #   OR: a page to fetch per row, instead of uri + content
      # sourceModifiedAt: "{{ edition }}"#   the version token; see below for the default
      sourceKind: bible                  # the store, generically; defaults to the realm name
      tags: [bible, kjv]                 # stamped on every document and chunk
    # extractEntities: false             # skip the per-document entity pass; see below (default true)
    sync:
      trigger: on-install                # on-install | scheduled | manual
      schedule: "0 0 3 * * SUN"          # six-field cron, host zone, for `scheduled`
      # watch:                           # a cheap read that changes when the records may have; see below
      #   cypher: MATCH (e:Edition) RETURN e.id AS edition, e.publishedAt AS at
      prune: true                        # remove documents no record produces any more

Records. from.cypher is read on behalf of the installing user, through the same scoping as that user’s own queries; a read the host cannot scope is a failed sync, never an unscoped one. from.urls gives one record per URL, exposed to the templates as url.

Two kinds of document. A record renders either text (document.uri + document.content: the realm renders the content itself, under a locator in its own scheme — the same uri contract as EXTERNAL_DOCUMENTS.md §3, so http(s)://, file:// and upload:// are refused) or a page (from.urls, or document.url per row: the host fetches and converts it, and the URL is its identity). Never both.

Versions decide what is re-ingested. Each document carries a version token, compared with the one held before anything is fetched or embedded:

	Token
document.sourceModifiedAt declared	the rendered value
rendered text, none declared	a hash of the rendered content — an unchanged record is never re-embedded
a page, none declared	the Last-Modified it is served with; a page that states none is re-read on every run

Triggers. on-install runs a source when the world has never completed it, when its declaration has changed since, or when its last run did not complete — after the realm’s reference/ data is seeded, so a cypher: source reading it never sees an empty graph. scheduled runs it on its cron, registered and removed with the realm. manual runs it only when asked. A run is always in the background; a second trigger while one is running is reported, not queued.

Watching for changes. A source whose records are expensive to read but rarely change declares sync.watch: a cypher: read, cheap to run, whose rows change whenever the records may have — a project’s commit, an edition’s date. The host reads it about once a minute for every world that has the realm installed and loaded, and runs the source when the rows differ from those of its last COMPLETE run, so a change is picked up within a minute or two whatever the trigger. A scheduled or on-install run whose watch is unchanged ends there, reading nothing else: it is reported as UNCHANGED and leaves the last run’s outcome in place. A run that is asked for (below), a changed declaration, and a last run that did not complete always read every record. A watch that cannot be read changes nothing and is tried again at the next check. The watch is only a trigger: each record’s version token still decides what is embedded, so a watch that changes more often than the records costs a read, never an embedding.

    sync:
      trigger: on-install
      watch:
        cypher: |                        # one row per project: changes when a commit lands
          MATCH (ip:IndexedProject)-[:AT]->(v:ProjectVersion)
          RETURN ip.key AS project, v.version AS version
      prune: true

realm_status reports, beside a watched source’s last run, checkedAt: when a watch last found it current.

Asking for a run. Any source, whatever its trigger, runs now when the user whose world installed the realm asks: realm_sync_documents (realm, and optionally one source) over MCP, or from an app through the app runtime’s gateway, gateway.realms.realmSyncDocuments({realm, source}). It is the user’s own world, read with the user’s own scoping, so no developer mode is needed, and only new or changed documents are embedded. An app that records something a source reads — a project to index, a page to keep — asks for the run straight after, rather than leaving the record unread until the next schedule:

await gateway.repository.createEntry({ type: 'IndexedProject', data: { key } });
await gateway.realms.realmSyncDocuments({ realm: 'polyglot-code', source: 'code-declarations' });

The run’s progress (RUNNING, with records, processed, landed, ingested) and outcome are under documentSources in realm_status, which an app reads as gateway.realms.realmStatus({realm}); an app declares both namespaces in its manifest ("namespaces": ["repository", "realms"]).

Pruning removes documents this source holds that no record produced, and only after a COMPLETE run: an empty read or a failed record deletes nothing, because “the source returned nothing” and “the source is gone” are indistinguishable from the host’s side. Nor does a read that says part of the source could not be read — a project whose repository could not be fetched, a source that refused its key or was unreachable: the run is PARTIAL, names what was not read, and keeps every document it held, so the source answering again re-embeds nothing. A watched source does not re-read every minute while that part stays unreadable; its watch changes when it can be read again, and the source runs.

Entities. Like any ingested document, each one a source lands is read once more after it is stored, by a model that finds the people, organizations and other entities it mentions and links the document to the ones the graph already holds: one model call per document landed. A source whose records already are graph entities — code declarations, a catalogue read from reference/ — gains nothing from that pass, so it declares extractEntities: false and its documents are stored, chunked and embedded with no model call. The default is true, and a world that has turned the pass off for all documents keeps it off whatever a source declares.

Status. Each run’s outcome — COMPLETE, PARTIAL (some records failed; retried next trigger), EMPTY (no records; nothing pruned), FAILED (with the reason) — is recorded per world and reported with the realm’s status, alongside every declared source that has never run.

Visibility. Documents are held per user and context, like every other ingested document: each world that installs the realm ingests its own copy. visibility: public is refused for a documents source until a shared document corpus exists.

When to write a handler instead. Declarative sources cover content the host can read or fetch unaided. A source behind OAuth, with a change feed, or needing a multi-step export (a Drive folder, a Notion workspace) still uses a handler and ctx.ingest.* (EXTERNAL_DOCUMENTS.md §6).

Validation reads sources.yml as strictly as a producer. A value outside a field’s vocabulary (visibility: publc, ordering: version-order) is an error at that field — reported, never quietly read as the default, because the default is the dangerous reading: unordered switches off the bias detection an ordered source needs, and private hides a label meant to be public. A source missing its name is an error at its own path, and does not take the rest of the file with it. A misspelled key is a warning that names the key it was probably meant to be.