MABTU crawlers

MABTU is a curated library of well-designed websites. Two small programs visit public websites on its behalf. If you saw one of them in your logs, this page explains what it did and how to stop it.

MABTU-Discovery

Reads design galleries (their sitemaps, feeds and listing pages) to learn which websites they featured, and makes one request to a featured site’s home page to check it is live.

• User agent: Mozilla/5.0 (compatible; MABTU-Discovery/1.0; +https://mabtu.com/bot)
• Obeys robots.txt for the MABTU-Discovery token and *, including Crawl-delay. Galleries that block AI crawlers are not read at all.
• At least 4 seconds between requests to the same host. Only pages it has not read before.
• Stops at any bot check. It never tries to get past one.

MABTU-Capture

A real browser that takes screenshots of a site’s public pages (home page first, then up to ten menu pages on desktop and mobile) and records design facts: colours, fonts, layout and technology.

• User agent ends with MABTU-Capture/1.0 (+https://mabtu.com/bot).
• Obeys robots.txt for the MABTU-Capture token and *.
• Declines optional cookies, fills in no forms, logs in nowhere, and collects no personal data.
• Paced per hosting provider with an hourly budget. After repeated refusals it pauses for hours rather than retrying.
• A site is captured again at most every few months.

Opting out

To stop both, add this to your robots.txt:

User-agent: MABTU-Discovery
Disallow: /

User-agent: MABTU-Capture
Disallow: /

To have a site already in the library removed, with its screenshots and data, use the removal form. Every listed site is credited and links to the original.