Re:Show HN: Crawlee – Web scraping and browser automation li...
by wumpus on Tuesday August 23, 2022 @01:08PM
(#32567893)
Most mentions of crawl-delay in robots.txt set a limit so slow that the website can't be fully crawled before the heat death of the Universe. That's why Google and bing etc. ignore crawl-delay.
[ Parent
| on HN ]
There are no comments on this comment yet.