5.6 Typed extraction — kind: extract (lazy entities)

Where an aggregate reduces an anchor’s neighborhood to ONE record, kind: extract fans it OUT: one model call per anchor extracts a list of typed records — the (d:Document)-[:HAS_CLAUSES]->(c:Clause) shape. The defining property is what the records are:

Extracted records are entities — part of the model, created lazily. The first traversal extracts and commits them; from then on they are ordinary graph, returned by a regular Cypher query with no engine involved.

With cache: {kind: graph} (the intended mode), the first traversal commits, per record:

• a real, scope-stamped node carrying the target label and __Entity__ — so extracted records surface wherever entities do (entity views, canonical KG queries), not only through the producing join;
• a display name — the extraction prompt must emit one per record (the record’s own heading, or a composed fallback like <category> §<section>); a nameless entity is invisible to entity surfaces, and the engine logs a warning when records arrive without one;
• a real containment edge (anchor)-[:edgeType]->(record) — committed right after the query’s read transaction closes (the open materialization transaction can hold locks on the anchor, so the edge write queues and lands the moment those locks release), owner-guarded on both ends so identical keys under two owners never cross-link; a hit whose containment is missing (a crashed earlier run) is healed on the spot;
• any declared record-to-record links (links:) — e.g. a clause whose references cites a sibling’s section commits (citing)-[:REFERS_TO]->(cited), document-local by construction.

# producers/clauses.yml (realm-authored — the host ships no domain prompt)
- name: clauseExtraction
  kind: extract
  edgeType: HAS_CLAUSES
  collect:
    targetLabel: Chunk
    via: PART_OF|HAS_PARENT*   # structured docs nest chunks under sections — traverse the containment path
    incoming: true
    anchorLabel: Document
    anchorKeyProperty: uri
    text: "{{ text }}"
  extract:
    role: workhorse            # a portable ROLE — never a concrete model name
    prompt: |
      Extract every clause as {name, category, text, section, references, sourceIndex} …
  links:
    - relationship: REFERS_TO
      fromField: references
      toField: section
  cache: { kind: graph }

The aggregate rules carry over with extraction-specific twists:

• Freshness is the shared input hash (collected items + the extract fingerprint, including the prompt and the links declaration). On change the anchor’s whole stale set is REPLACED — extraction counts can shrink, so stale rows are DETACH DELETEd, taking their containment and link edges with them; the fresh set re-commits nodes and edges together. Changing the realm’s prompt migrates every anchor’s set the same way, on next traversal.
• Vocabulary is enforced, not requested. A oneOf validation rule on a target-type property (e.g. the category taxonomy) is a hard gate: the engine DROPS any extracted record that violates it before persisting — prompt discipline alone does not hold this line.
• Honest-empty is a transient miss: nothing extractable ⇒ nothing persisted, nothing deleted, retried next traversal.
• Steering stays transient — a {ai: {hint: …}} or model override extracts fresh with transient identities and never touches the committed canon.
• Demand applies across anchors — a cold LIMIT 1 over many documents extracts from the first anchor(s) only, until produced records satisfy the budget.
• Extraction cost tracks the missing work, not document size. Extraction over a document proceeds in bounded batches, each cached independently. A repeat traversal over unchanged content makes no model calls — including for portions that extracted to zero records. Editing part of a document re-extracts only the affected portion; the rest of the committed record set is untouched. A cancelled extraction keeps its completed work; re-asking resumes rather than restarts.
• Partial materialization converges. A LIMIT n traversal over cold content may stop extracting as soon as n records exist, leaving the committed set partial; any later, broader traversal completes it by extracting only what is still missing. Whatever sequence of asks produced it, the set converges to the same records a single full extraction of today’s content would produce. Declared record-to-record links (links:) appear as soon as both ends exist; until then the citing field still records the claim. The engine’s cost estimate for a partially extracted document counts only the remaining work — a query refused as too expensive can become allowed after part of the corpus has been materialized (e.g. via a narrowed opt-in ask).

Because the containment edge is real, RETURN c returns the entity itself — prefer it over scalar projections when the caller wants the records rather than a report about them.

5.6.1 cardinality: — reshaping a document into ONE record

Not every extraction fans out. A filing, a certificate, a form or a statement is one record of a declared type, and declaring that is not cosmetic: it changes what the engine guarantees.

- name: registrationReshape
  kind: extract
  cardinality: one            # one | optional | many (default: many)
  edgeType: HAS_REGISTRATION
  collect: { … }              # unchanged
  extract:
    fromType: true            # the field list comes from the target type (§5.6.2)
    prompt: |
      The excerpts below come from a company registration filing. Prefer the registrar's own spellings.
  cache: { kind: graph }

MATCH (d:Document)-[:HAS_REGISTRATION]->(r:CompanyRegistration)
RETURN d.title, r.companyNumber, r.incorporationDate, r.status

Value	One anchor yields	Zero records means
many (default)	0..N records — today’s fan-out	honest-empty, retried
one	exactly one record	the declaration was wrong or the source is unreadable — honest-empty, and reported
optional	0..1 records	an ordinary answer (not every document is a filing)

What one/optional guarantee, beyond the count:

• One row per anchor, always. The record’s identity is the anchor’s, so a re-read replaces it. A document can never accumulate two registrations, and “the registration” is never a coin toss between generations.
• Fields scattered through the document assemble into one record. A long source is read in parts; each part reports only what it shows, and the parts are combined. Where two parts state the same field differently, the earlier statement wins and the disagreement is reported rather than silently resolved. A list-valued field (a filing’s officers) unions across parts instead of stopping at the first.
• All-or-nothing. If any part of the source could not be read, nothing is written and nothing is returned: half a form is wrong, not incomplete. The next ask re-reads it. For the same reason, a LIMIT never truncates a single record mid-way — it stops between anchors, never inside one.
• Freshness is whole-document. A single record spans the whole source, so no part of it is fresh on its own: any change re-reads the form. (The many regime’s part-by-part incrementality does not apply, and would have nothing to serve until the whole form had been read anyway.)
• Declared types hold on the stored record. A property declared int/number/boolean is stored as that type, not as the wording the source used: 1,240 is a number you can compare with >, and $120,000.50 does not become 120. This holds for the record as later read by ORDINARY Cypher, not only through the join — which is the point, since the record becomes plain graph after first materialization. (This applies to every producer kind, not only extract and not only the single regime — any fetched property its type declares is stored as that type, a declared list element by element; it simply matters most here, where a form’s fields are dates, counts and amounts rather than prose.)
• A “not stated” answer is silence, not a value. A source that states nothing for a field leaves it absent; the record never carries N/A, unknown or a placeholder as though it were data, and a source that states nothing at all yields no record rather than an empty shell.

Everything else in §5.6 is unchanged: records are entities, the containment edge is real, declared vocabularies are enforced, steering stays transient.

5.6.2 fromType: — the type states the shape, the prompt states the domain

With fromType: true the record’s field contract — names, types, which are lists, and any declared oneOf vocabulary — is taken from the target type’s own declaration rather than restated in the prompt. The prompt is then free to say only what the type cannot: what the document is, and what to prefer when it is ambiguous.

This closes a gap that is otherwise silent. A typed extraction that spells its fields out in prose states its shape twice — once for the model to read, once as properties:/validation: for the engine to enforce — and the enforcing copy always wins. A vocabulary that has drifted out of the prompt is one the model is never told about, whose records are dropped on arrival, and the symptom is an empty result with no error. Derived, the two cannot disagree.

Default is off, so existing extractions are unaffected. Turning it on changes the extraction and re-reads each anchor once, exactly as a prompt edit does.

The excerpts themselves are supplied whether or not the prompt renders them, so a prompt that carries only domain guidance is a complete prompt. (A field list with no text to read is the one input that reliably makes a model invent a plausible record.)

---