Re:Show HN: HTML-to-Markdown – convert entire websites to Ma...
by kgeist on Saturday November 09, 2024 @06:33AM
(#42093852)
I remember there was a paper which found that LLMs understand HTML pretty well, you don't need additional preprocessing.
The downside is that HTML produces more tokens than Markdown.
[ Parent
| on HN ]
Re:Show HN: HTML-to-Markdown – convert entire websites to Ma...
by simonw on Saturday November 09, 2024 @06:48AM
(#42093909)
Right: the token savings can be enormous here.
Use https://tools.simonwillison.net/jina-reader to fetch the https://news.ycombinator.com/ homepage as Markdown and paste it into https://tools.simonwillison.net/claude-token-counter - 1550 tokens.
Same thing as HTML: 13367 tokens.