Re:Show HN: Irchiver, your full-resolution personal web archive
by MerelyMortal on Saturday December 04, 2021 @04:41PM
(#29444552)
None of those websites say they do what I'm asking. Did you try them out? (How do you know?) If they do what I'm asking, their websites aren't that good if I can't figure that out by reading them.
Do an `nb $URL` in the CLI and `nb` will save the data as a markdown file in a collection managed by `git`. You can then e.g. `nb search` to find what you're looking for, or `nb browse` to see a rendered version of the markdown in a browser of your choice.
Re:Show HN: Irchiver, your full-resolution personal web archive
by lambdaba on Friday December 03, 2021 @06:11PM
(#29436794)
Oh, true, I missed that.
In chrome it would be quite trivial to tail the history read from the sqlite db and get the latest url, and frankly something like Archivebox would be more powerful (truly grabbing all resources). Unfortunately in Firefox it's not possible to tail the history without unlocking the database, so the solution would be slightly more hackish but still feasible.
For illustration, this is the script I use to grab Chrome history:
profile=$HOME/.config/google-chrome/Default
sqlite3 "file:$profile/History?immutable=1" \
"select datetime(last_visit_time/1000000-11644473600, 'unixepoch'), url from urls order by last_visit_time asc"
It would be pretty easy to use `watch` or `inotifywait` to do this regularly and call Archivebox with the new urls.
Re:Show HN: Irchiver, your full-resolution personal web archive
by MerelyMortal on Friday December 03, 2021 @08:01PM
(#29437620)
It pulls URLs from your history, and downloads them later. It doesn't download what you see, so if you're logged in somewhere, it won't be able to save that; or if content changes between when you see it and it gets scraped, it won't know.
Re:Show HN: Irchiver, your full-resolution personal web archive
by tedmiston on Monday December 06, 2021 @11:14PM
(#29469000)
The use case you raise makes a lot of sense, especially on sites with paywalls or where you have paid accounts.
I haven't explored this route personally yet, but it may be possible to copy cookies from your browser cookie jar over if that would satisfy your use case.
(I imagine a Firefox extension could probably do this.)
I see lots of things where you have to click-to-save, but nothing automatic.
https://archivy.github.io/
https://perkeep.org/
https://github.com/go-shiori/shiori
Are three of them
⌘ https://github.com/xwmx/nb
Do an `nb $URL` in the CLI and `nb` will save the data as a markdown file in a collection managed by `git`. You can then e.g. `nb search` to find what you're looking for, or `nb browse` to see a rendered version of the markdown in a browser of your choice.
Note: this bundles external resources like images, css and scripts inside the file
In chrome it would be quite trivial to tail the history read from the sqlite db and get the latest url, and frankly something like Archivebox would be more powerful (truly grabbing all resources). Unfortunately in Firefox it's not possible to tail the history without unlocking the database, so the solution would be slightly more hackish but still feasible.
For illustration, this is the script I use to grab Chrome history:
It would be pretty easy to use `watch` or `inotifywait` to do this regularly and call Archivebox with the new urls.I haven't explored this route personally yet, but it may be possible to copy cookies from your browser cookie jar over if that would satisfy your use case.
https://github.com/ArchiveBox/ArchiveBox/wiki/Configuration#...