Re:Lightpanda: The headless browser designed for AI and auto...
by andrethegiant on Saturday January 25, 2025 @01:10PM
(#42823348)
This is why I’m making crawlspace.dev, a crawling PaaS that respects robots.txt, implements proper caching, etc by default.
Also, throttling requests and following robots.txt actually makes it less likely that your scraper will be blocked, so even for those who don't care about the ethics, it's a good thing to have ethical defaults.