News for hackers. Stuff that matters.
Re:Show HN: HTML-to-Markdown – convert entire websites to Ma... by kgeist on Saturday November 09, 2024 @06:33AM (#42093852)
I remember there was a paper which found that LLMs understand HTML pretty well, you don't need additional preprocessing. The downside is that HTML produces more tokens than Markdown.
Re:Show HN: HTML-to-Markdown – convert entire websites to Ma... by simonw on Saturday November 09, 2024 @06:48AM (#42093909)
Right: the token savings can be enormous here.

Use https://tools.simonwillison.net/jina-reader to fetch the https://news.ycombinator.com/ homepage as Markdown and paste it into https://tools.simonwillison.net/claude-token-counter - 1550 tokens.

Same thing as HTML: 13367 tokens.