News for hackers. Stuff that matters.
Re:How to Crawl the Web with Scrapy by matheusmoreira on Monday September 13, 2021 @03:51PM (#28515936)
Yeah. I hate cloudflare, captchas. Why can't these companies accept that our scrapers are valid user agents? Only Google is allowed to do it, nobody else.
Re:How to Crawl the Web with Scrapy by Eikon on Monday September 13, 2021 @04:15PM (#28516267)
Because most scrapers aren't providing any value to website owners, in fact, they are costing them, unlike google.