News for hackers. Stuff that matters.
Re:47% of all internet traffic came from bots in 2022 by acdha on Sunday May 14, 2023 @04:22PM (#35941077)
I’d believe half of all traffic to non-video hosts, however. We provide a lot of free content and it’s staggering how many bad robots there are - not actual malice, just what you’d expect from lousy programmers who don’t get the bill. Things like the same IP / User-Agent downloading the same file thousands of times just in case it changed in the last 5 seconds or crawling millions of permutations of search parameters rather than using the site maps. Many week that’s half of the total traffic, many tens of terabytes of HTML & JSON.
Re:47% of all internet traffic came from bots in 2022 by Akronymus on Sunday May 14, 2023 @04:37PM (#35941177)
As someone who once wrote a scraper to interface with a site that lists lawyers (So we can blacklist them in advance from being called, because they REALLY don't like getting contacted by accident), and another time a scraper to get a list of unemployed people (employee leasing): Bad bots really ruin it for those that actually try to make well behaved scrapers.
Re:47% of all internet traffic came from bots in 2022 by acdha on Sunday May 14, 2023 @09:03PM (#35942835)
Yes. Stuff like the default Python user agent being blocked by Cloudflare cause hassles for students & are entirely due to sloppy bot developers.