Windy

The web crawler behind Semiote's search engine. If it has been to your site, this page is what the request pointed at.

User agent

SemioteWindy/0.1 (+https://windy.semiote.org/crawler)

robots.txt token

SemioteWindy

Contact

abuse@semiote.org

Operated by

Semiote

What it takes

Per site: robots.txt, the home page, and — only if the home page looks like a site that has been maintained — pages linked from it or listed in its sitemap, the stylesheets those pages reference, and one icon.

It never fetches JavaScript. Not a file, not a bundle, not a source map. There is no browser engine involved and nothing is executed. Script elements are removed from the markup before it is stored.

It also never requests images, video, fonts, archives or documents, never submits a form, never sends a cookie back, and never authenticates. If your site is not one it wants, it costs you two requests and does not come back for three months.

How to block it

Ordinary robots.txt, obeyed as RFC 9309 describes it — the longest matching rule wins and Allow breaks a tie.

User-agent: SemioteWindy
Disallow: /

Or part of it:

User-agent: SemioteWindy
Disallow: /private/
Allow: /private/press/

Naming Semiote instead blocks every crawler we will ever run. A robots.txt that returns a server error is treated as a full stop rather than as permission: if your server cannot state its rules, we leave it alone until it can.

How to slow it down

User-agent: SemioteWindy
Crawl-delay: 10

Honoured. Without it we wait two seconds between requests and keep one connection open at a time per host, however much work there is. A 429 or 503 stops us on the whole host, not just the one address, and Retry-After is obeyed.

Who to write to

abuse@semiote.org, which is also sent as the From header on every request, so it is in your access log beside whatever prompted you to look. A person reads it.

How to check a request is really ours

Anyone can copy a user agent, and scrapers borrow reputable ones routinely. To confirm:

• Reverse-look up the address that connected. It resolves to a host under crawl.semiote.org.
• Forward-look up that host. It resolves back to the same address.

Both steps are needed; the second is what makes it unforgeable. Anything claiming to be Windy that fails this is not Windy, and we would like to know about it.