Trafilatura: Python tool to gather text on the Web

Re:Trafilatura: Python tool to gather text on the Web by d4rkp4ttern on Tuesday August 15, 2023 @07:21AM (#37132545)

I’ve been using this package and like it a lot.

One problem I’d like to find a solution for is how to get past cookie pop ups when scraping a website. I’ve not found a satisfactory packaged solution for this. Clearly a tough problem in general but wondered if people have found good libs to help with this. I’ve heard of solutions involving playwright etc.

[ Parent | on HN ]

Re:Trafilatura: Python tool to gather text on the Web by adbarba on Tuesday August 15, 2023 @08:02AM (#37132875)

Thanks! Here is what I put together in the docs, you could basically preprocess/render/filter the webpages with the software of your choice and then pass the result to trafilatura: https://trafilatura.readthedocs.io/en/latest/troubleshooting...

[ Parent | on HN ]