Re:Trafilatura: Python tool to gather text on the Web
by adbarba on Tuesday August 15, 2023 @05:38AM
(#37131905)
Author here, nice to see the package on the HN's front page this morning and thanks for the kind words! Just created an account to participate in the discussion, I'll try to answer your questions.
[ Parent
| on HN ]
Re:Trafilatura: Python tool to gather text on the Web
by d4rkp4ttern on Tuesday August 15, 2023 @07:21AM
(#37132545)
I’ve been using this package and like it a lot.
One problem I’d like to find a solution for is how to get past cookie pop ups when scraping a website. I’ve not found a satisfactory packaged solution for this. Clearly a tough problem in general but wondered if people have found good libs to help with this. I’ve heard of solutions involving playwright etc.
Re:Trafilatura: Python tool to gather text on the Web
by adbarba on Tuesday August 15, 2023 @08:02AM
(#37132875)
Thanks! Here is what I put together in the docs, you could basically preprocess/render/filter the webpages with the software of your choice and then pass the result to trafilatura: https://trafilatura.readthedocs.io/en/latest/troubleshooting...