Running web sites makes me not want to

Re:Running web sites makes me not want to by iririririr on Thursday November 12, 2026 @12:07PM (#50470590)

thinking running a website never got you 99.9% of requests from bot is just delusional.

anyway, if you really want to block scrappers, give up Anubis. it just about real users and do nothing against bots. simple block ISN that belong to cloud providers. it's that simple.

[ Parent | on HN ]

Re:Running web sites makes me not want to by dhosek on Thursday November 12, 2026 @12:22PM (#50470815)

I got to watch traffic shift from cloud IPs to residential proxies in real time after I blocked the cloud IPs. It’s endless whack-a-mole.

What really pisses me off was Meta’s bots ignoring robots.txt when I tried to prevent high levels of traffic hitting meaningless URL variants on a Mediawiki site. Everybody else stopped hitting those URLs but not Meta which was flooding me with so much traffic it was impacting site performance. There source IPs and user agents are now banned.

[ Parent | on HN ]

Re:Running web sites makes me not want to by viraptor on Thursday November 12, 2026 @01:27PM (#50471867)

Do not explicitly block that traffic. Return 200 with fake content or 404 and they won't try to workaround it.

[ Parent | on HN ]

Re:Running web sites makes me not want to by threetonesun on Thursday November 12, 2026 @12:36PM (#50471055)

Some of us deployed websites before bots. Heck once upon a time you had to ask nicely for them to even come visit.

[ Parent | on HN ]