In that context, I was referring to this iocaine.
- 0 posts
- 4 comments
- interestingasfuck@lemmy.world•I hid a link on my website that only bots can see and told them not to follow it. They broke the rules and in 4 days they went 124,414 levels deep into an endless maze. One bot opened 782,580 doors.…byalgernon@lemmy.ml1 month
Most crawlers don’t care - they’re purely downloading. Most garbage will also be filtered out before training, and generating plausible garbage is expensive - not really worth the trouble. What you can do with the garbage, is that you can place unique identifiers in the links you serve, so when a crawler that you can’t recognize otherwise comes visiting a poisoned URL, you can firewall it off. This works because the crawlers use a queue: they get a list of URLs to visit, they download them, and later on, the URLs from those get extracted, and crawlers will visit those URLs.
So if you manage to trap a bot, and overwhelm its queue with poisoned URLs, you can also catch bots that you wouldn’t be able to easily identify otherwise. And that is why serving an endless maze of garbage is so very useful. Because it lets you catch stuff like headless chromes operating through residential proxies: because those too, follow a queue.
Technology@lemmy.zip•Aggressive AI scrapers are making it kinda suck to run wikisbyalgernon@lemmy.mlEnglish
6 monthsIt can stop them nowadays, by firewalling some of the crawlers off. The reason it doesn’t stop them by default is because it serves them poisoned URLs, which it can later identify if the crawlers come back riding a headless Chrome. But once they do that, and hit a poisoned URL, there’s little reason to let them wander in an endless maze further: serve one request, and block the IP.
I’ve been running that on my own infra, and my daily number of requests went down from ~50+ million to… 2 million.



I did! It was lucky I found it so soon, I usually do a search once or twice a week - and it just happened I did one of those yesterday.
That’s normal, iocaine’s name is from the Princess Bride, after all - and the book and the film are much better known than my software (as they should be).
The ingested poison doesn’t do much, it’ll get discarded before training. So if you have nothing to put behind iocaine, I don’t think it’s worth it to run it. It can generate endless amounts of garbage, yeah, and will tie up some crawler resources too, yup, but it also costs you bandwidth and CPU & RAM you otherwise wouldn’t need to spend (since there’s nothing behind it, there’s no “more expensive” endpoint iocaine is taking a hit for). I don’t think that’s a good trade. As for Proof of Work: most bots won’t bother, and for the few that do… the cost/benefit ratio is still bad if there’s nothing behind iocaine.