Show HN: I made a tool to clean and convert any webpage to Markdown

Re:Show HN: I made a tool to clean and convert any webpage t... by scary-size on Monday April 15, 2024 @03:02AM (#40037718)

Thoroughly scraping is challenging, especially in an environment where you don’t have (or want) a JavaScript runtime.

For content extraction, I found the approach the Postlight library takes quite neat. It scores individual html nodes based on some heuristics (text length, link density, css classes). It the selects the nodes with the highest score. [1] I ported it to Swift for a personal read later app.

[1] https://github.com/postlight/parser

[ Parent | on HN ]

Re:Show HN: I made a tool to clean and convert any webpage t... by justech on Monday April 15, 2024 @04:13AM (#40038016)

This is pretty cool. Care to share your Swift port?

[ Parent | on HN ]

Re:Show HN: I made a tool to clean and convert any webpage t... by scary-size on Monday April 15, 2024 @04:32AM (#40038118)

Not planning to. It’s my first Swift/iOS project. I neither want to polish it nor maintain it publicly. Happy to share it privately, email is in the bio. I’m planning on a blog post describing the general approach though!

[ Parent | on HN ]

Re:Show HN: I made a tool to clean and convert any webpage t... by rismay on Monday April 15, 2024 @04:23PM (#40045130)

Care to share the Swift port?

[ Parent | on HN ]

Re:Show HN: I made a tool to clean and convert any webpage t... by Kikobeats on Saturday April 20, 2024 @06:40PM (#40101653)

For getting the HTML, you can use microlink, just passing the URL to https://html.microlink.io/{url}, like https://html.microlink.io/https://example.com

[ Parent | on HN ]