News for hackers. Stuff that matters.
Re:Show HN: Crawlee for Python – a web scraping and browser ... by Findecanor on Tuesday July 09, 2024 @11:47AM (#40917468)
Does it have support for web scraping opt-out protocols, such as Robots.txt, HTTP and content tags? These are getting more important now, especially in the EU after the DSM directive.
Re:Show HN: Crawlee for Python – a web scraping and browser ... by jancurn on Tuesday July 09, 2024 @12:22PM (#40917919)
Not yet, but it’s on the roadmap