News for hackers. Stuff that matters.

A comment on So you want to Scrape like the Big Boys? (2021). Parent comment

Re:So you want to Scrape like the Big Boys? (2021) by eli on Saturday April 27, 2024 @04:24PM (#40183148)
Isn’t that a choice the website owner should be able to make?
[ Parent | on HN ]
Re:So you want to Scrape like the Big Boys? (2021) by jMyles on Saturday April 27, 2024 @05:02PM (#40183523)
Of course it's your choice to make.

Is someone forcing you to respond to requests you'd prefer to ignore?

[ Parent | on HN ]
Re:So you want to Scrape like the Big Boys? (2021) by paulryanrogers on Saturday April 27, 2024 @06:14PM (#40184078)
If crawlers are stealth DDoSing my site then I lose the ability to respond entirely.
[ Parent | on HN ]
Re:So you want to Scrape like the Big Boys? (2021) by theamk on Saturday April 27, 2024 @07:01PM (#40184388)
yes, people like OP who get the farms of scrapers.

The website owners make their preferences clear with robots.txt, IP blocks and other antibot technology. Scrapers intentionally ignore owners' desires and force the to respond.

[ Parent | on HN ]