Inside Wayback Machine, the internet’s time capsule

Re:Inside Wayback Machine, the internet’s time capsule by bane on Sunday September 30, 2018 @09:15PM (#18109363)

Anybody know of something like this that I can use for personal archiving?

[ Parent | on HN ]

Re:Inside Wayback Machine, the internet’s time capsule by xj9 on Sunday September 30, 2018 @09:49PM (#18109478)

https://www.archiveteam.org/index.php?title=Wget_with_WARC_o...

  #!/bin/sh

  cd ${HOME}/data/archive

  user_agent="Mozilla/5.0 (Windows; U; Windows NT 6.1; en-US) AppleWebKit/533.20.25 (KHTML, like Gecko) Version/5.0.4 Safari/533.20.27"

  domain="${1}"

  wget \
    --mirror \
    --page-requisites \
    --restrict-file-names=nocontrol \
    --timeout 60 \
    --tries 5 \
    --wait 1 \
    --waitretry 5 \
    --warc-cdx \
    --warc-file="${domain}" \
    --warc-header "operator: Wayback Mesh" \
    -U "${user_agent}" \
    -e robots=off \
    "http://${domain}"

  ipwb index "${domain}.warc" > "${domain}.ipfs.cdxj"

[ Parent | on HN ]

Re:Inside Wayback Machine, the internet’s time capsule by telotortium on Friday October 12, 2018 @04:40AM (#18200107)

https://www.gwern.net/Archiving-URLs

[ Parent | on HN ]

Re:Inside Wayback Machine, the internet’s time capsule by fake-name on Monday October 01, 2018 @03:39AM (#18110684)

https://github.com/fake-name/ReadableWebProxy

Disclaimer - It's a project of mine, specifically written for my interests, and is very much not production ready.

It does have a full blown spider in it, which supports distributed fetching and stuff via another project [1].

It does both rewritten archiving (basically overwriting the page style) and raw archiving.

I also have a bunch of miscellaneous stuff like a custom python interface for chromium so I can handle jerberscript shitshows (cloudflare "protection" [2], reCaptcha [3], etc...). People seem really dead-set on ruining the internet.

I keep meaning to implement WARC proxying to feed into the IA as a parallel output stream for the spider, but it's a lot of work and I have too many projects already.

1: https://github.com/fake-name/AutoTriever 2: https://github.com/fake-name/ChromeController 3: https://github.com/fake-name/WebRequest

[ Parent | on HN ]

Re:Inside Wayback Machine, the internet’s time capsule by PeterMikhailov on Sunday September 30, 2018 @11:26PM (#18109820)

https://wallabag.org/en

If you run your own, make sure you turn on "Download images"

[ Parent | on HN ]

Re:Inside Wayback Machine, the internet’s time capsule by zaarn on Monday October 01, 2018 @03:09AM (#18110565)

Wallabad isn't quite suitable for Archival operation last I checked since it doesn't present or save the original webpage.

That's one of the major points why archive.is/wayback is popular; you get the exact same page as you get in the browser. Perfectly for archival.

[ Parent | on HN ]

Re:Inside Wayback Machine, the internet’s time capsule by nikisweeting on Monday October 01, 2018 @10:59AM (#18113112)

The self-hosted way-back machine!

Bookmark Archiver: https://github.com/pirate/bookmark-archiver

It's modeled on the internet archive except it uses Chrome Headless in addition to wget so you get snapshots of the page after JS executes too.

[ Parent | on HN ]