Skip to content
Embabel Worlds

5.16 elasticsearch — a search index as a relevance source

An elasticsearch producer joins an Elasticsearch index or alias — self-managed or hosted — by RELEVANCE: the anchor’s text is the query, each hit becomes a virtual node linked to the anchor it was found for, and the retrieval provenance rides on the edge. It is the relevance sibling of sparql/cypher: the realm declares WHICH cluster, index and retrieval; no query author writes a search body.

- name: contractSearch
  kind: elasticsearch          # alias: elastic
  index: contracts             # an index or alias; wildcards allowed (docs-*)
  connection:
    endpoint: "https://cluster.example.org:443"
    apiKeyEnv: ACME_ES_API_KEY # env var; never a credential in YAML. Omit for an open cluster
    timeoutSeconds: 30
  retrieval:
    mode: hybrid               # hybrid (default) | bm25 | semantic | knn
    fields: [title, body]      # the lexical leg
    semanticField: body_semantic   # a self-embedding field; required by every mode but bm25
    rankWindowSize: 50
    rankConstant: 60
    rerankInferenceId: my-reranker   # optional; needs rerankField
    rerankField: body
  hits:
    k: 5
    minScore: 0.0
    idField: contractNumber    # optional: a business key as identity, instead of the document id
    project: { counterparty: "party.name" }
  cache: { kind: ttl, seconds: 300 }

Query it like any relevance edge, selecting it with via where a target is reachable more than one way:

MATCH (me:AssistantUser)-[:TRACKS]->(t:ResearchTopic)
MATCH (t)-[rel:RELEVANT_TO {via: 'elasticsearch'}]->(d:IndexedDocument)
RETURN d.title, rel.score, rel.docId, rel.index
ORDER BY rel.score DESC

The contract:

  • One search per anchor, composed cluster-side. Hybrid retrieval is a single request the cluster plans and fuses — not several searches merged afterwards — so a fused ranking is the cluster’s, and k is what comes back per anchor.
  • hybrid is the default and fuses lexical with semantic by reciprocal rank. bm25 is purely lexical and needs no semantic field, no inference and no model access — the mode that runs against any cluster. semantic and knn query a self-embedding field by TEXT: the index owns the embedding, so nothing is vectorized outside the cluster.
  • Every hit is citable. Each match carries score, docId, index, mode and rank on the EDGE, not on the node. So a document relevant to two anchors keeps a distinct score and rank for each — and an answer built on retrieved documents can cite the exact document in the exact index it came from.
  • Identity is the document id, or a business key you name. With hits.idField declared, that field IS the identity and a hit lacking it is DROPPED and reported — never silently keyed by the document id instead, which would split one document across two identities. The document id remains available as docId provenance either way.
  • minScore filters at the source. Fused (RRF) scores are small and not comparable to BM25 scores; leave the floor at 0 for hybrid unless you have measured the distribution.
  • Reranking is opt-in. A declared reranker re-scores the retrieval it wraps. It is the one stage that requires a deployed model, which is why it is off by default.
  • Honesty. A failed search — network, HTTP error, unparseable or non-search response — is a warnings entry plus an empty result, never a silent 0 that reads as “you have no such documents”. A rejected key (HTTP 401) is reported as an authentication problem, distinct from a generic failure. A declared apiKeyEnv that is unset is reported BEFORE the cluster is called, because an unset variable and a refused key have different fixes. A retrieval that cannot be composed as declared — a semantic mode with no semanticField, a reranker with no field — is reported by name as an authoring error rather than degraded to a weaker search that would return plausible, quietly worse results.

Cost every consumer should know: one live search per anchor, against a cluster the realm does not control. A fan-out over many anchors is many searches — bound it with a narrowing match before the hop, and declare a TTL cache for repeated questions.