Re:Web Scraping for Me, but Not for Thee
by dvhh on Friday August 25, 2023 @10:37PM
(#37269453)
There are some factors that might lead to this lone of though.
- it seem a lot of web developer / product manager either do not know or do not care about robots.txt
- some web application are so badly optimized, that some of them are not able to handle more than 1 hit per second at a sustained rate, which admittedly have worked fine so far. But crawlers are persistent, causing the normal crawling activity to cause denial of service for normal users.
- it seem a lot of web developer / product manager either do not know or do not care about robots.txt
- some web application are so badly optimized, that some of them are not able to handle more than 1 hit per second at a sustained rate, which admittedly have worked fine so far. But crawlers are persistent, causing the normal crawling activity to cause denial of service for normal users.