News for hackers. Stuff that matters.
Re:Show HN: HTML-to-Markdown – convert entire websites to Ma... by oezi on Saturday November 09, 2024 @06:14AM (#42093778)
Does it also include logic to download JS-driven sites properly or is this out of scope?
Re:Show HN: HTML-to-Markdown – convert entire websites to Ma... by simonw on Saturday November 09, 2024 @06:54AM (#42093924)
It doesn't. For that you would need to execute a full headless browser first, extract the HTML (document.body.innerHTML after the page has finished loading can work) and process the result.

If you're already running a headless browser you may as well run the conversion in JavaScript though - I use this recipe pretty often with my shot-scraper tool: https://shot-scraper.datasette.io/en/stable/javascript.html#... - adding https://github.com/mixmark-io/turndown to the mix will get you Markdown conversion as well.

Re:Show HN: HTML-to-Markdown – convert entire websites to Ma... by JohannesKauf on Saturday November 09, 2024 @06:54AM (#42093925)
That is unfortunately out of scope. I like the philosophy of doing one thing really well.

But nowadays—with Playwright and Puppeteer—there are great choices for Browser automation.

Re:Show HN: HTML-to-Markdown – convert entire websites to Ma... by jot on Saturday November 09, 2024 @09:38AM (#42094649)
We do that with Urlbox’s markdown feature: https://urlbox.com/extracting-text
Re:Show HN: HTML-to-Markdown – convert entire websites to Ma... by bni on Saturday November 09, 2024 @10:23AM (#42094883)