Skip to content
Embabel Worlds

5.10 feed — an RSS/Atom feed as a keyed source

Some sources publish no JSON API — they publish a search feed (ParlInfo’s Hansard search, media-release feeds, gazettes). A feed producer makes one a join target: each anchor key becomes one search, each feed item one record (title, url, publishedAt, summary), stamped with the key under keyAs.

- name: estimatesMentions
  kind: feed
  url: 'https://parlinfo.aph.gov.au/parlInfo/feeds/rss.w3p;query=Dataset:estimate Content:"{key}"'
  redirectMeansEmpty: true      # this source signals ZERO RESULTS with a redirect
  keyAs: searchPhrase
  maxItems: 8
  minIntervalMs: 1000
  cache: { kind: ttl, seconds: 3600, negativeTtlSeconds: 3600 }

URL template. {key} is the whole anchor key, percent-encoded. Composite keys split on | into {key1}..{key9}, and a part can carry a DATE FORMAT: {key2|date:dd/MM/yyyy} parses that part as an ISO date/instant and renders it in the source’s own format — so a date-ranged search (Date:{key2|date:dd/MM/yyyy} >> {key3|date:dd/MM/yyyy}) is first-class, never string-smuggled. A non-ISO date part fails that key loudly rather than searching for garbage.

Guarantees

  • publishedAt is an ISO instant whenever the feed’s date parses (RFC-1123 with numeric offsets included); an unparseable date passes through raw rather than being dropped.
  • redirectMeansEmpty: true maps a redirect to an HONEST EMPTY, never a failure and never a followed link — for sources that signal “no results” by redirecting to an error page. Without it, redirects are followed normally.
  • Every failure is a diagnostic plus zero rows — an unreachable feed, a non-XML body, an HTTP error can never read as “no mentions”.
  • A capped read says so — hitting maxItems reports a truncation.

follow: — the item’s PAGE, reduced to phrase-anchored excerpts. With follow: { maxPages: 3, match: '{key1}', excerptChars: 4000 } the producer fetches the first maxPages item pages, strips them to text, and keeps ONLY bounded windows around matches of the match term (a key template, so a composite key’s phrase part anchors the excerpt, never its date parts) into a content property. The whole page is never stored; a followed page that never mentions the term contributes no content; a page that cannot be read is reported as UNKNOWN content — never as “the term does not occur”. The same holds for an item past maxPages: its page was never read, so its content is UNKNOWN, not empty. The answer says so — it is marked partial, with a FOLLOW_CAPPED note naming the budget, how many items went unread, and the levers (raise maxPages up to its ceiling, or narrow the query) — so a WHERE m.content CONTAINS … over those items is never mistaken for a complete search. A filter never answers for a page nobody opened. A WHERE that reads a followed field (m.content IS NULL, NOT m.content CONTAINS …, as much as CONTAINS) leaves the unread items out rather than judging their missing text, and the note says how many were left out and that an aggregate over the filtered rows is a LOWER bound. A query that only projects the field keeps every item, with the field null where its page went unread, and the note says whose value is unknown — a count() of the rows stands, a count/collect over the field is a lower bound. Speaker-attributed transcript prose survives inside the windows, which is the point: summarize(m.content, '…') over followed Hansard fragments can honestly answer “what was discussed, and by whom”. Follow knobs are ceiling-clamped (pages ≤ 5, excerpt ≤ 8000 chars) and page fetches ride the same pace gate as the feed itself.

A rogue realm cannot consume ridiculous resources — by construction, not by convention. The engine enforces ceilings a spec may tighten but never exceed, rejected at LOAD time: maxItems ≤ 200; at most 25 anchor keys per fetch (the excess is dropped and REPORTED); a fixed byte cap on every response (5MB); fixed connect/read timeouts; sequential fetching with a pace floor of 500ms between requests (minIntervalMs may slow a producer, never speed it past the floor). A feed source is someone’s search endpoint, not a bulk API.