AMDmi3
For the fun of it, I'm developing my own anti-bot solution which acts as a proxy between nginx and the application, and is able to modify and override application responses (I anubis works the same way). What I've tried so far:
- Introducing a tarpit - infinite amount of cheap dynamically generated interlinked garbage pages, a hidden nofollow+noindex link to which is added to legitimate pages. - Ability to introduce artificial response delays to tarpit or other pages. - Ban IPs accessing a tarpit (and some other broken path patterns which browsers will never visit). - Set a simple boolean cookie to know whether session has already visited the site. - Set another boolean cookie for clients accessing a tarpit (in case bots learn to store cookies or reuse sessions; not tried to use it). - For heaviest application endpoints (which need protection the most), require either "already visited" cookie, or show simple captcha. - Also show captcha to banned IPs.
Each of these seems to help to some extent, and there's a lot of room for both experiments and for the case where bots adapt to some of these mechanisms.
Findings: - Tarpit seems to divert a lot of traffic from legitimate pages. - However, bots understood how well it performs and increased request rate from 40 to 160 rps. - Introducing a delay helped with that. 9 sec delay has brought the rate down to 40 rps immediately. Longer delay makes bots timeout and drop the connection - not useful in my case as I want to feed more garbage links to them. - It suggests the important idea of that the botnet has some centralized rate control mechanism which adapts to site performance, so you just need to make the site look slow and underperforming to bots. - Most basic cookie based logic works great (for now). No need for cryptography or complex captchas, just set already_visited=1, or passed_captcha=1, where "captcha" is just an automated js which sets a cookie and reloads the page. - IP bans work too, but simple `HashMap<IpAddr>` does not scale. Will try to come with more clever solution someday, probably involving a tree of IP address space regions, or a registry of nets from WHOIS, gathers statistics on per-network basis and acts based on that. For instance, nets with no legitimate requests over large period of time may be banned on firewall to reduce server load. Other nets with high rate of bot hits may be considered untrusted and always require captcha. - Botnets ignore a `rel="nofollow noindex"`, so you can probably safely use these to guide the botnet to tarpit, while legitimate crawlers should not be affected.
Bot-resistant application design wisdom: - Obviously, make it fast. You can no longer afford the application to perform slow (over 100ms) SQL queries. That's ok for humans, but bots will crush you. You can no longer afford slow backend language. I've rewritten repology to rust a couple of years ago, which resulted in around 10-100x performance and memory optimization. Had I not done it, the site would already be down by now. - Don't ignore dynamic solutions. I myself am an oldschool who prefers to generate HTML on backend and avoid javascript. Guess it's time to reconsider. In my case, doing pagination in js would a) hide most pages from the botnet b) avoid generating infinite amount of distinct pagination pages, while these pages being heaviest on the site. - Nginx before the application currently consumes more CPU than the application and anti-bot solution combined, so I'm considering retiring it.
Questions to community: - I'm not satisfied with tarpit pages with just links to other tarpit pages. Any ideas of feeding the botnet with actual content to poison the dataset? I can come up with fake API reference pages with nonexistent methods.