Underminer aims to discourage AI model crawler bots and other unwanted bots, while allowing search engine crawlers and RSS aggregators to work normally.
It works in two ways:
- Using
robots.txtrules to prohibit well-behaved AI training bots from crawling your site. - By selectively corrupting the meaning of text content, to sabotage badly-behaved bots that ignore those rules.
Well-behaved search engine bots are permitted to crawl the site as normal, without being restricted by robots.txt or being subjected to sabotage.
Well-behaved LLM training bots that obey robots.txt will not receive any corrupted content.
Features
- 100% free and open source. No upsells or nags or subscriptions or promotions or freemium versions or donations. Ever.
- Auto-detects crawler bots.
- Completely invisible to real users and well-behaved bots.
- Verifies IP ranges of well-known bots to detect fake Googlebot crawlers etc.
- (Almost) zero-configuration.
- Language-neutral; works with page and post content in most languages.
- Preview mode: you can see what a bad bot will see.
Instead of blocking bad bots, they will simply be served corrupt and useless content:
- Words and sentences are randomly rearranged.
- Lists and paragraphs are re-ordered.
- Numbers are randomized.
- Currency symbols and measurement units are randomly changed.
- Alt text and descriptions are randomly switched around.
- Links are switched around.
- Image URLs are intentionally broken.
Well-behaved bots
For the purposes of this plugin, a well-behaved bot is one that:
- Obeys
robots.txt. - Publishes an up-to-date list of IP ranges.
- Identifies search crawling separately from AI model training crawling.
Requirements
- PHP 8.0 or higher
- WordPress 6.4 or higher
