Re:Reading the web offline and distraction-free
by genewitch on Monday October 11, 2021 @07:09AM
(#28827003)
Firefox headless can save webpages as images or "as a complete page" can't it?
I use the image functional and only sites like Twitter fail to work correctly, although it's probably my CGI gateway timing out waiting for JavaScript or whatever.
Re:Reading the web offline and distraction-free
by tasogare on Sunday October 10, 2021 @04:04PM
(#28821534)
That's a solved problem: click right, save the page as MHTML. It can be opened later and online with a browser again.
If I recall correctly the only gotcha is that the option to save in this format needs to be enable using the flags settings.
The take-any-webpage-offline need is also common in the education space (teachers want to save a webpage and send it to their students as part of a lesson and don't want to worry about availability or ads etc).
I used to work on tools for this https://github.com/learningequality/ricecooker/blob/develop/... and https://github.com/learningequality/BasicCrawler/blob/master... which worked quite well for most sites, but still very far from a general-purpose solution.
There is also more powerful/general-purpose scraper that generates a ZIM file here: https://github.com/openzim/zimit
It would be really nice to a "common" scraper code base that takes care of scraping (possibly with a real headless browser) and outputs all assets as files + info as JSON. This common code base could then be used by all kinds of programs to package the content as standalone HTML zip files, ePub, ZIM, or even PDF for crazy people like me who like to print things ;)