sources.yml — declaring how a COLLECTION behaves

producers/ say how to fetch records. sources.yml says what the collection is: how big, what it can be filtered by, what order it arrives in, how often it changes, and whether it should be mirrored at all. The platform cannot infer any of that, and getting it wrong is not a performance problem — it is a correctness one.

The failure that motivated this: a register of ~426,000 planning applications, filterable only by council, returned oldest-first with no sort option. Reading it with a page cap looked prudent and was silently catastrophic — the records outside the cap were always the most RECENT ones, so a surface reporting “no application on this lot” was omitting exactly the applications a user was asking about. Nothing in a producer spec could have expressed that hazard.

sources:
  - name: nsw-planning-register
    producer: applicationsByCouncil        # the producer that reads it
    label: PlanningApplication             # the node label it materializes

    shape: bounded                         # bounded | unbounded | per-key
    cardinality: 426000                    # order of magnitude is enough
    partition: council                     # the ONLY axis the source can filter
    queriedBy: [address, street, lot, date, cost]   # axes users ask on and the source CANNOT filter
    ordering: oldest-first                 # unordered | newest-first | oldest-first | irrelevant
    updates: daily

    sync:
      strategy: mirror                     # mirror | lazy | live
      trigger: on-first-use                # on-first-use | on-install | scheduled | manual
      refresh: full-rewalk                 # incremental | full-rewalk
      watermark: DateLastUpdated           # the field that dates a record
      watermarkFilterable: false           # can the SOURCE filter on it? here: no

    completeness:
      declaredTotal: TotalCount            # response field giving the true size of a partition
      aggregatesRequireComplete: true      # withhold derived statistics on partial coverage

    visibility: public                     # public | org | private