Wayback machine gets a facelift, new features

Re:Wayback machine gets a facelift, new features by Cameron_D on Monday November 04, 2013 @07:13PM (#6672145)

You can probably make your service do both screenshots and WARC, instead of loading a site directly, load it through WARC Proxy (https://github.com/odie5533/WarcProxy), that will write out a WARC file and you can still store your screenshot.

Once you have the WARCs you can upload them to Archive.org and they can be added to the wayback, or you can set up your own service for browsing them, built off something like warc-proxy https://github.com/alard/warc-proxy (Yeah, same name different purpose...)

There is also a MITM version of WARCProxy that will let you store HTTPS sites: https://github.com/odie5533/WarcMITMProxy

[ Parent | on HN ]

Re:Wayback machine gets a facelift, new features by hncommenter13 on Monday November 04, 2013 @07:20PM (#6672179)

As of version 1.14, wget natively supports warc (including built-in gzip and cdx index file generation).

http://www.archiveteam.org/index.php?title=Wget_with_WARC_ou...

This makes creating a browse-able mirror of a site in warc format fairly straightforward, as wget will automatically make links relative, as well as fetch requisite files (css, js, images) for each page.

[ Parent | on HN ]

Re:Wayback machine gets a facelift, new features by Cameron_D on Monday November 04, 2013 @07:23PM (#6672195)

Yeah, but as far as I can guess, derwiki's service doesn't use wget, so running a proxy to store the WARCs is the next-simplest thing.

[ Parent | on HN ]

Re:Wayback machine gets a facelift, new features by toomuchtodo on Monday November 04, 2013 @09:50PM (#6672916)

If his service runs on any sort of Linux distro, its stupid simple to call wget with a system call. Wget comes standard with all of the most popular distros.

[ Parent | on HN ]