Wayback machine gets a facelift, new features

Re:Wayback machine gets a facelift, new features by derwiki on Monday November 04, 2013 @04:00PM (#6671131)

I just launched a similar service called https://www.DailySiteSnap.com that screenshots, emails, and archives a specified web site on a daily basis. My use case is to be able to look back at any one day and see what my site looks like, since Archive.org doesn't refresh my page as often as I update it.

Disclaimer: I'm really not trying to over-market myself, but I figured readers of this thread might be interested in my project. Happy to take down this post if it's read as too spammy.

[ Parent | on HN ]

Re:Wayback machine gets a facelift, new features by toomuchtodo on Monday November 04, 2013 @04:04PM (#6671152)

You can always package your site up in a format Archive.org can read:

http://www.archiveteam.org/index.php?title=Wget_with_WARC_ou...

http://www.archiveteam.org/index.php?title=The_WARC_Ecosyste...

http://justsolve.archiveteam.org/wiki/WARC

http://warcreate.com/

[ Parent | on HN ]

Re:Wayback machine gets a facelift, new features by derwiki on Monday November 04, 2013 @04:07PM (#6671176)

Thanks for the info, was unaware of ARC/WARC formats. That said, I still think many people are looking for something simpler/easier, and a daily screenshot is good enough. Particularly, it will guarantee preserved formatting as browsers continue to evolve.

[ Parent | on HN ]

Re:Wayback machine gets a facelift, new features by Cameron_D on Monday November 04, 2013 @07:13PM (#6672145)

You can probably make your service do both screenshots and WARC, instead of loading a site directly, load it through WARC Proxy (https://github.com/odie5533/WarcProxy), that will write out a WARC file and you can still store your screenshot.

Once you have the WARCs you can upload them to Archive.org and they can be added to the wayback, or you can set up your own service for browsing them, built off something like warc-proxy https://github.com/alard/warc-proxy (Yeah, same name different purpose...)

There is also a MITM version of WARCProxy that will let you store HTTPS sites: https://github.com/odie5533/WarcMITMProxy

[ Parent | on HN ]

Re:Wayback machine gets a facelift, new features by hncommenter13 on Monday November 04, 2013 @07:20PM (#6672179)

As of version 1.14, wget natively supports warc (including built-in gzip and cdx index file generation).

http://www.archiveteam.org/index.php?title=Wget_with_WARC_ou...

This makes creating a browse-able mirror of a site in warc format fairly straightforward, as wget will automatically make links relative, as well as fetch requisite files (css, js, images) for each page.

[ Parent | on HN ]

Re:Wayback machine gets a facelift, new features by Cameron_D on Monday November 04, 2013 @07:23PM (#6672195)

Yeah, but as far as I can guess, derwiki's service doesn't use wget, so running a proxy to store the WARCs is the next-simplest thing.

[ Parent | on HN ]

Re:Wayback machine gets a facelift, new features by toomuchtodo on Monday November 04, 2013 @09:50PM (#6672916)

If his service runs on any sort of Linux distro, its stupid simple to call wget with a system call. Wget comes standard with all of the most popular distros.

[ Parent | on HN ]

Re:Wayback machine gets a facelift, new features by toomuchtodo on Monday November 04, 2013 @04:16PM (#6671230)

I agree that screenshots are easier; WARC files are future-proof.

[ Parent | on HN ]

Re:Wayback machine gets a facelift, new features by d0n on Monday November 04, 2013 @04:45PM (#6671411)

That would be nice, if you mailed the screenshots. But it looks like you just send a link to the screenshot.

[ Parent | on HN ]