The Zetlyn crawler

Zetlyn is a search engine that builds a relationship graph — it measures verifiable properties of public web pages and links pages that support or contradict each other. Our crawler fetches public pages to do this.

How to identify it

Requests come with a User-Agent of the form:

Zetlyn/1.0 (+https://zetlyn.com/crawler; abuse@zetlyn.com)

How we crawl

• We obey robots.txt (access rules) and honor TDM reservations (tdm-reservation header, tdmrep.json, in-page opt-outs).
• We rate-limit per host and back off on 429/503.
• We store measured properties, not your full text — cached HTML is kept only to re-measure and is never republished.

How to block or limit it

Add to your robots.txt:

User-agent: Zetlyn
Disallow: /            # block entirely
Crawl-delay: 10        # or just slow us down

To reserve content from text-and-data-mining, publish a TDM reservation (tdmrep.json) or send tdm-reservation: 1.

Contact

Abuse, takedown, or crawl questions: abuse@zetlyn.com.
Security: see /.well-known/security.txt.

Zetlyn records how pages relate — who cites, quotes, duplicates or shares a subject with whom, and which sites are one operator. It never judges whether sources agree. If a relation is wrong, it can be reported for review.