neo1

Just an aside. mawk 2.0 (high performance awk) has been out for about a decade. It's about 2x the performance of its predecessor mawk 1.3.4, and 4x performance of GNU awk. Practically no distro uses it, because the author Mike Brennan is not the "I maintain this code in perpetuity" guy and the mawk maintainer Thomas E. Dickey doesn't care move on from from 1.3.4.

The point is: Code needs to be maintained if it is to be relied on. I haven't heard good things about LLM's on that front.

Defuddle looks interesting, I will try it out. Thanks for pointing it out!

There is another tool that on the surface appears similar to Defuddle called trafilatura: https://github.com/adbar/trafilatura It was OK, but I wasn't happy with it, as it kept stripping the comments sections.

It's called w3m (or lynx). No seriously. Not kidding. While I'm trying to be snarky, I recently discovered how well it works to do exactly what you are asking. Sites that need javascript, or are paranoid cloudflare prisons won't work, but otherwise it works quite well.

For walls of text (long substack articles) I still prefer epub, and use https://codeberg.org/neo1/mkebook which I wrote, but if you love the terminal: w3m. Filters are done by your rss reader, newsboat has boat loads of options, but any decent one should.

Seems like a good place to plug: https://codeberg.org/neo1/mkebook

it's a bash script front end for curl-cffi -> pandoc i.e. impersonate web-browser to download url, then convert to epub. There is also the option to use chromium for the scraper or ebook-convert for the converter. Some config and you have a (rss reader) newsboat -> calibre with a hotkey.