Sources

A source is a directory with one declaration in it. Fetching, paging, remembering where it got to, indexing, and answering what it can be asked are the source's own job, and none of it is written twice.

Seven kinds of source
A declaration: what one claim is
The source says how it wants to be read
What every update does, and refuses to do

1. Seven kinds of source

Where claims come from differs; what they look like afterwards does not.

Type	What it reads
`folder`	Files on disk, or a git checkout it keeps up to date. Text is extracted from what it finds; a `.json` file is structured and reachable by field.
`csv`	A file or a URL, with its delimiter. One row is one claim unless the declaration says otherwise.
`xlsx`	A sheet by name, a header row, and the same treatment from there.
`http`	A list endpoint, a detail endpoint, or both. Paging by offset, by page number, or by the `Link` header.
`feed`	RSS and Atom: the first that fetches from other people's servers, and the reason for the pacing rules below.
`sql`	A query against PostgreSQL, rows as JSON, with `{since}` as its watermark. The connection string must be a variable, never the file.
`hub`	Subscribed rather than fetched. The claims arrive built, so there is nothing to extract and nothing to re-run: an update asks its hub for a newer version.

Two more are specified and not built: MySQL, and a folder of images whose claims carry their metadata. GitHub is http, written once: --from github:advisories, a repository's releases or advisories, or its checkout.

2. A declaration: what one claim is

Five prefixes say where a value comes from: field: from the source, file: from the file itself, meta: from its metadata, text: from the extracted text, const: from the declaration. A field has a type, and a value that will not parse is reported by the update rather than quietly dropped.

A claim needs an identifier, written as identified_by at the top of the file, and that is what a tracker joins on. It is stored as the source wrote it and compared with the case folded, because one publisher's CVE-2021-44228 is another's cve-2021-44228 and neither of them is wrong.

# sources/kev/source.yaml
name: zetlyn/cve-kev
title: CISA Known Exploited Vulnerabilities
kind: vulnerability
fetch:
  type: csv
  path: https://www.cisa.gov/sites/default/files/csv/known_exploited_vulnerabilities.csv
schedule:
  every: 1h
identified_by:
  cve: field:cveID
claims:
  title: field:vulnerabilityName
  text:
  - field:vulnerabilityName
  - field:shortDescription
  known: field:dateAdded
  properties:
    # Membership is the claim. Every row of this catalogue is a vulnerability
    # CISA has evidence of being exploited, so the property is a constant and
    # not a column: the ransomware question below is a different one.
    exploited:
      type: code
      from: const:yes
    due_date:
      type: date
      from: field:dueDate
    known_ransomware_campaign_use:
      type: code
      from: field:knownRansomwareCampaignUse
views:
- name: recent
  default: true
  columns: [due_date, known_ransomware_campaign_use, known]
  sort: known desc
retention:
  history: true   # the default, said out loud

3. The source says how it wants to be read

Its columns, its facets, its sort, its named views and the example queries that prove they work. A tracker adopts what a source declares rather than inventing a second opinion about somebody else's data, and a source with columns nobody else has keeps them: an exploit is not a vulnerability claim with empty fields.

4. What every update does, and refuses to do

Watermark	Where the last complete update got to. A partial update does not advance it, so what it failed to reach is fetched again rather than skipped forever.
Partial	An update that did not finish removes nothing and claims nothing. An empty answer from a slice of time is not a fault.
Shape	40% fewer claims than the store holds, or, on an update that read the whole source, a property every claim used to carry now missing: the transaction is rolled back and the update reads `refused` with the reason.
Pacing	A 429 or a 503 doubles the whole update's pause, to a ceiling of twenty seconds, and it eases back after fifty quiet answers.
History	Kept by default, for good: every value a claim ever had, what the source handed over for it, and when. A claim can be read as it stood on a date, and a change is reported property by property with the old value beside the new.

And one command reads the declaration against the store: zetlyn source check. An example that returns nothing, a column naming a field no claim carries, a promise that no longer holds. None of it is wrong until somebody reads it, which is why an update never catches it. It exits non-zero when it finds any, so a CI stops the change that would have published it.

Next: the calls a source and a tracker answer, or what a tracker composes from sources.