Re:Show HN: Crawlee for Python – a web scraping and browser ...
by Findecanor on Tuesday July 09, 2024 @11:47AM
(#40917468)
Does it have support for web scraping opt-out protocols, such as Robots.txt, HTTP and content tags?
These are getting more important now, especially in the EU after the DSM directive.
[ Parent
| on HN ]