Zetlyn
A source

Point it at a source.  It handles the rest.

A source is a directory with one declaration in it. Fetching, paging, remembering where it got to, indexing, and answering what it can be asked are the source's own job, and none of it is written twice.

WHERE CLAIMS COME FROM

Seven kinds of source,
one shape afterwards.

TYPE

  • folderFiles on disk or a git checkout it keeps up to date. Text is extracted from what it finds; a .json file is structured and reachable by field
  • csvA file or a URL, with its delimiter. One row is one claim unless the declaration says otherwise
  • xlsxA sheet by name, a header row, and the same treatment from there
  • httpA list endpoint, a detail endpoint, or both. Paging by offset, by page number, or by the Link header
  • feedRSS and Atom, which is the first that fetches from other people's servers and the reason for the pacing rules below
  • sqlA query against PostgreSQL, rows as JSON, with {since} as its watermark. The connection string must be a variable, never the file
  • hubSubscribed rather than fetched. The claims arrived built, so there is nothing to extract and nothing to re-run: an update against one of these asks its hub for a newer version

Two more are specified and not built: MySQL, and a folder of images whose claims carry their metadata. GitHub is http, written once: --from github:advisories, a repository's releases or advisories, or its checkout. Saying which is which is cheaper than finding out later.

A DECLARATION

What one claim is.

Five prefixes say where a value comes from: field: from the source, file: from the file itself, meta: from its metadata, text: from the extracted text, const: from the declaration. A field has a type, and a value that will not parse is reported by the update rather than quietly dropped.

A claim needs an identifier, written as identified_by at the top of the file, and that is what a tracker joins on. It is stored as the source wrote it and compared with the case folded, because one publisher's CVE-2021-44228 is another's cve-2021-44228 and neither of them is wrong.

field: file: meta: text: const:
sources/kev/source.yaml
name: zetlyn/cve-kev
title: CISA Known Exploited Vulnerabilities
kind: vulnerability
fetch:
  type: csv
  path: https://www.cisa.gov/sites/default/files/csv/
        known_exploited_vulnerabilities.csv
schedule:
  every: 1h
identified_by:
  cve: field:cveID
claims:
  title: field:vulnerabilityName
  text:
  - field:vulnerabilityName
  - field:shortDescription
  known: field:dateAdded
  properties:
    # Membership is the claim. Every row of this catalogue is a vulnerability
    # CISA has evidence of being exploited, so the property is a constant and
    # not a column: the ransomware question below is a different one.
    exploited:
      type: code
      from: const:yes
    due_date:
      type: date
      from: field:dueDate
    known_ransomware_campaign_use:
      type: code
      from: field:knownRansomwareCampaignUse
views:
- name: recent
  default: true
  columns: [due_date, known_ransomware_campaign_use, known]
  sort: known desc
retention:
  history: true   # the default, said out loud
WHO OWNS THE VIEW

The source says how
it wants to be read.

Its columns, its facets, its sort, its named views and the example queries that prove they work. A tracker adopts what a source declares rather than inventing a second opinion about somebody else's data, and a source with columns nobody else has keeps them: an exploit is not a vulnerability claim with empty fields.

A RUN

What it does,
and what it refuses to do.

EVERY RUN

  • watermarkWhere the last complete update got to. A partial update does not advance it, so what it failed to reach is fetched again rather than skipped forever
  • partialAn update that did not finish removes nothing and claims nothing. An empty answer from a slice of time is not a fault
  • shape40% fewer claims than the store holds, or, on an update that read the whole source, a property every claim used to carry now missing: the transaction is rolled back and the update reads refused with the reason
  • pacingA 429 or a 503 doubles the whole update's pause, to a ceiling of twenty seconds, and it eases back after fifty quiet answers
  • historyKept by default, for good: every value a claim ever had, what the source handed over for it, and when. A claim can be read as it stood on a date, and a change is reported property by property with the old value beside the new

And one command that reads the declaration against the store: zetlyn source check. An example that returns nothing, a column naming a field no claim carries, a promise that no longer holds. None of it is wrong until somebody reads it, which is why an update never catches it. It exits non-zero when it finds any, so a CI stops the change that would have published it.

ZETLYN

Data in. The source does the rest.

The interface What is served