Show HN: I made a tool to clean and convert any webpage to Markdown

Re:Show HN: I made a tool to clean and convert any webpage t... by msp26 on Monday April 15, 2024 @05:45AM (#40038447)

Thanks for the links I had no idea those existed.

For my article web scraper (wip) the current steps are:

- Navigate with playwright + adblocker

- Run mozilla's readability on the page

- LLM checks readability output

If check failed

- Trim whole page HTML context

- Convert to markdown with pandoc

- LLM extracts from markdown

[ Parent | on HN ]

Re:Show HN: I made a tool to clean and convert any webpage t... by privatenumber on Monday April 15, 2024 @10:39AM (#40041158)

Mozilla has released Readability as a standalone package so you can avoid spinning up a browser entirely: https://github.com/mozilla/readability

[ Parent | on HN ]

Re:Show HN: I made a tool to clean and convert any webpage t... by msp26 on Monday April 15, 2024 @11:57AM (#40042259)

I still wanted the browser for UBlock Origin and handling sites with heavy JS. I was using the standalone Readability script already but today I ended up dropping it for Trafilatura. It works a lot better.

The inefficiency of using a browser rather than just taking the html doesn't really matter because the limiting factor is the LLM here.

And yes the LLM is essential for getting clean data. None of the existing methods are flexible enough for all cases even if people say "you don't need AI to do this".

[ Parent | on HN ]

Re:Show HN: I made a tool to clean and convert any webpage t... by asadalt on Monday April 15, 2024 @11:00AM (#40041459)

you would still need to run. For js based websites.

[ Parent | on HN ]