Re:Serve Markdown to AI Agents with Accept Headers
by honr on Thursday August 27, 2026 @11:51AM
(#49466820)
I found HTML itself works [slightly] better with some LLMs (that I happen to frequent). So, when scraping, I parse the resulting html, simplify it, and produce simpler html contents (closer to semantics of what I guessed the real content was). Given processing tools for html are far more mature, I tend to keep content in semantic, low structure html.
That's effectively using an LLM as an HTML to markdown converter, which is both absurdly wasteful and also surprisingly inexpensive (if you use a model like GPT-5.6 Luna.)