It’s a constant fight. It’s also the reason why there is that “verifying your browser…” message when loading MB sometimes. That’s one of those anti-AI-scrapper measures.
Robots.txt only works for nice bots that respect it.
Prompt injection seems like a nice idea but if you are dealing with actors ruthless enough to use thousands of residential IPs and other means to disguise themselves as humans they likely also have taken precautions to prevent exactly that.
These scrapers already purposefully disguise their IP addresses. They are not going to respect robots.txt rules.
edit: ninja’d by chaban, but worth reiterating I think! Basically, there’s no simple solution, or our (very excellent) server admins would have done it.
It will not for most AI scrapers we are facing, because they are designed to avoid any restriction (and that’s the business model of the companies selling those services: scrape unlimited).
That said, it doesn’t cost much to add, especially for AI agents that actually care, for example, they can use MB web service, instead of MB website in many cases…).
