News for hackers. Stuff that matters.

A comment on 47% of all internet traffic came from bots in 2022. Parent comment

Re:47% of all internet traffic came from bots in 2022 by buro9 on Sunday May 14, 2023 @02:25PM (#35940120)
It's a multi-tenant platform (about 300 forums, with the biggest being around 250K visitors per month), the database is clearly now on a vertically scaled box that is now too excessive but I was able to delete a few of the Linodes that were horizontally scaling the API and Web UI (the Web UI is just a client of the API hence those could be saved too).

I've also noticed that my cache hit rate is extraordinary now, which I assume is because humans read recent stuff and bots read the long-tail of old stuff.

[ Parent | on HN ]
Re:47% of all internet traffic came from bots in 2022 by mike_d on Sunday May 14, 2023 @03:17PM (#35940556)
As someone who does targeted scraping of forums, I can say having a good open API and caching is probably the best way to decrease load.

If you use Cloudflare, turn off their anti-bot stuff. It is far more efficient to let them just serve bots from the cache than having scrapers use tricks to bypass them and go directly to your origin server.

[ Parent | on HN ]
Re:47% of all internet traffic came from bots in 2022 by Spivak on Sunday May 14, 2023 @07:48PM (#35942437)
Yeah there's basically no way I would ever read what you just wrote and choose to do anything but block harder even at higher cost.

Bots end running around CF would guarantee turning on authenticated origin pull.

[ Parent | on HN ]