Skip to content
Embabel Worlds

5.9 Range partitioning — partition: on a remote producer

A producer whose anchor key is a DATE RANGE (<fromInstant>/<toInstant>, ISO instants) meets a hard limit on sources that cap one request: the page cap. A two-month window against a feed that returns newest-first and caps at maxPages * size records silently yields the newest slice — a result that LOOKS like a recent window, which is worse than an error. partition: fixes this at the source declaration:

- name: releasesPublishedInWindow
  kind: remote
  keyArgs: [from, to]
  echoKeyAs: publishedWindow        # REQUIRED with partition (see below)
  paging: { style: cursor, ..., maxPages: 12 }
  partition: { unit: day, maxUnits: 366 }

The engine splits each range key into per-unit sub-ranges (UTC-aligned; hour | day | week | month), fetches each sub-range as its own request — each with its own maxPages budget — and stamps every record back to the CALLER’s key.

Guarantees

  • Nothing about queries changes. The same MATCH on the same window key returns the same shape — just complete. The expansion is invisible to the join, the cache, and the query; records always link to the key the caller wrote, never to an engine-minted sub-range.
  • Union of parts = whole. Sub-ranges tile the window exactly (contiguous, half-open, the caller’s own from/to at the edges). A record a source returns on both sides of a boundary (inclusive-bounds sources) is deduplicated.
  • A key that is not a range — or spans a single unit — is fetched AS-IS. Declaring partition: never adds calls to a small request, and a mixed batch (an id and a window) behaves.
  • Truncation is per sub-range and NAMED. A day that still caps reports “truncated … for key ‘2026-06-30…/2026-07-01…’” — the caller knows which slice to re-ask, and every other day stays complete. COMPLETE now means “every sub-range exhausted its feed”, a claim with a proof.
  • maxUnits is a backstop, not a result-shaper. A range expanding past it keeps the MOST RECENT maxUnits sub-ranges and reports the uncovered head by name — loudly, never silently.
  • Overlapping ranges share sub-fetches. Two windows in one batch that share days fetch each shared day once; each caller’s records carry its own key.
  • Progress is per sub-range — a long scan ticks “day 18 of 61” on the events stream instead of an unmoving spinner.

emptyErrorPatterns — when a source says “no records” through an HTTP error. Some APIs answer an empty range with a failure status rather than an empty list (AusTender returns HTTP 400 "No Records found for Date Range"). Partitioning turns that from a rare edge case into a CONSTANT one — every quiet weekend is an empty day — and a warning storm buries the real diagnostics. Declaring the patterns on the producer maps a matching failure to an honest empty page: no diagnostic, no retry, no breaker; the walk simply ends there.

  emptyErrorPatterns: ["No Records found"]     # substring, case-insensitive

Declare it ONLY for text that unambiguously means zero results. Anything broader converts real failures into silent empties — the exact deception the rest of this spec exists to prevent.

Costs and rules

  • partition requires echoKeyAs: sub-fetches ask the source with keys the caller never wrote, so the source cannot echo the caller’s key — the stamp is the only honest link. A spec without it fails at load time.
  • A complete broad scan makes MORE calls (one-plus per unit) and takes proportionally longer. The producer’s declared cost.rate paces them; pair broad scans with a watched/background invocation rather than a bigger client timeout.
  • The realm picks unit from the source’s real volume: a feed of hundreds/day partitions by day; a firehose by hour; a sparse register by month. maxPages then only has to cover the busiest single unit.