News for hackers. Stuff that matters.
Re:Show HN: Crawlee – Web scraping and browser automation li... by wumpus on Tuesday August 23, 2022 @01:08PM (#32567893)
Most mentions of crawl-delay in robots.txt set a limit so slow that the website can't be fully crawled before the heat death of the Universe. That's why Google and bing etc. ignore crawl-delay.

There are no comments on this comment yet.