I simply use the browser's (Firefox) save page feature (ctl-s). Then every now and then, I convert the folder with these pages to a squashfs image (which de-duplicates all the CSS, JS, image files that are saved multiple times). I then use shell tools to search (ls grep locate etc.). This doesn't save the URL, but I also maintain a private "bookmarks" Git repository for more interesting bits where each bookmarked resource gets its own file (in a descriptive hierarchy), along with thoughts and notes.
What works well is that I am somewhat selective, complete trash doesn't end up being archived, it's pretty space efficient. It's also simple to wade through (in the shell), each saving action is just a file and folder pair. Also often I use the reader mode Firefox feature, and then save that (i.e. I save what I saw, not what was delivered). What doesn't work so well is that saving the page is often a bit of a hassle, often Firefox reports save errors then I just ctl-s again to see if the previous attempt succeeded, if not, hit the reload icon in the download task list which apparently forces it. Also, that it saves embedded videos, haven't figured out yet how to disable that, so periodically I remove videos again. Also, when saving multiple pages from the same site, links to other pages don't go to the local mirror. (In cases where it's important, I use wget -m -k.)
- You wouldn't have to bother with JS files, they are removed from saved pages by default.
- You couldn't (easily) de-duplicate resources because they are embedded in base64 into the page. However, SingleFile can detect all the hidden elements and unused CSS rules/declarations (by computing the cascade). So the HTML and CSS are optimized. It is also able to group duplicate images (by using CSS custom properties). There are other options to make the document size as small as possible. Most of the time, pages saved with SingleFile are smaller than Chrome MHTML files.
- The saving process would be much more simple, reliable and could be automatic.
- Videos wouldn't be saved (by default), a snapshot of the video would replace each video.
- Maybe the "filename template" in SingleFile would help you to organize things.
[1]https://chrome.google.com/webstore/detail/save-as-mht/hfmodl...
EDIT: it could be related to the fact that SingleFile is maybe more strict regarding the HTTP header "Content-Type". For example, it will discard images with "text/html" as content type value.
[1] https://blog.chromium.org/2018/10/trustworthy-chrome-extensi...
You could take this and post process your internet history on a rolling basis to accomplish your goal.
[1] https://github.com/gildas-lormeau/SingleFile