News for hackers. Stuff that matters.
Re:Screenshots Forever and Ever Until You Can’t Stand it by jmduke on Wednesday August 20, 2014 @01:24AM (#8201193)
I've been thinking a lot the past few weeks about the preservation of art and goods in general. I think it comes from two main sources:

1. Maciej Ceglowski (`idlewords) has been talking a lot recently about link rot. Recently, he found that 25% of items pinned a mere five years ago were dead links and 17% from only three years ago were dead [^1]. That's an incredibly high rate -- and, selfishly, one I'm noticing as my bookmark folders for recipes (RIP BroEats.com) and designs and other things is filled with more and more duds.

2. I've been reading Do Not Sell At Any Price [^2], a book about the subculture of 78rpms. These are records that are so rare and so -- for lack of a better term -- unwanted by the vast majority of the music-listening populace that the act of collecting them is less about hoarding and more about preservation. To quote one of the characters in the book (roughly from memory):

"It's a weird feeling, holding this thing in your hand and knowing that you could break the song," he said. "I snap this record in half and this song is lost forever. It's a lot of responsibility, and sometimes I think that's why I take it so seriously."

I can honestly say that, prior to reading this article, I had no idea what a ZX Spectrum was. Now, after some digging, I do -- and I still have no desire to play one, obtain one, or hold onto it in any meaningful way. (And seeing as I'm usually on the weirdly attached end of the spectrum with these kinds of things, I doubt I'm the only one.) But I'm struck by how important it is to hold onto these things, even if its in a cardboard box in a forgotten closet somewhere or a link on the Internet Archive that gets clicked once every couple decades.

I'm not positing that there will ever be a point in time that someone has the hankering to play ZX Spectrum Xtreme Chess, but I think there's inherent value in preserving this ecosystem -- something of a testament to the people who made it, the people who played it, the novelty that at one point in time there were five million living rooms with this machine in it.

The Web turned 25 this year, and it's already coming down with acute cases of memory loss. I'm hoping that by the time it hits fifty, the problem won't have gotten worse -- it will have gotten much, much better, not just with URLs but with remembering the time when people played 3D StarFighter by the Oliver Twins. [^4]

(This is a very roundabout way of saying the following: Jason, you are completely awesome for doing this, and thanks for sharing it with us.)

[^1] https://twitter.com/Pinboard/status/501406303747457024

[^2] http://www.amazon.com/Not-Sell-Any-Price-Obsessive-ebook/dp/...

[^3] The nice things about IA links is you can pretty reasonably assume that they won't suffer from link rot, right?

[^4] https://www.flickr.com/photos/textfiles/14771850314/in/set-7...

Re:Screenshots Forever and Ever Until You Can’t Stand it by ijk on Wednesday August 20, 2014 @02:15AM (#8201280)
> The nice things about IA links is you can pretty reasonably assume that they won't suffer from link rot, right?

The catch there being, the Internet Archive retroactively respects robots.txt that forbid crawling, so if someone gets control of a domain they can block the archived pages. This is a big problem with lapsed domains that get swept under the umbrella of a holding company that has lots of domains pointing to the same content, with a blanket robots.txt.

Re:Screenshots Forever and Ever Until You Can’t Stand it by ggreer on Wednesday August 20, 2014 @04:28AM (#8201516)
This is a huge problem. Sites like NASA's NTRS are retroactively blocked.[1] It's not clear which user agents one must allow in robots.txt. NTRS allows archive.org_bot, but apparently ia_archiver is also needed. At some point the allow directive in the NTRS robots.txt[2] no longer matched, nuking all historical data.

1. See http://web.archive.org/web/20121029225832/http://ntrs.nasa.g... for an example.

2. http://ntrs.nasa.gov/robots.txt

Re:Screenshots Forever and Ever Until You Can’t Stand it by db48x on Wednesday August 20, 2014 @08:25PM (#8205385)
It's not been nuked, merely hidden.
Re:Screenshots Forever and Ever Until You Can’t Stand it by jleader on Wednesday August 20, 2014 @02:56AM (#8201343)
A few years ago I talked to an IA engineer, who said they were planning on dealing with this by not crawling sites whose nameservers were known to point to a domain parking company. The idea was that if they never retrieved the robots.txt, they wouldn't retroactively apply it. I don't know if that filtering out of parking nameservers ever happened, and it wouldn't help for parked domains whose robots.txt they'd already retrieved, but but it would help with domains that lapse in the future.
Re:Screenshots Forever and Ever Until You Can’t Stand it by StavrosK on Wednesday August 20, 2014 @05:32AM (#8201660)
But why retroactively remove the data? The original owner was fine with holding it, why should the snapshot be deleted because a completely different person wants his completely different website to not be crawled?
Re:Screenshots Forever and Ever Until You Can’t Stand it by bjt on Wednesday August 20, 2014 @12:50PM (#8203128)
It's hard for a bot to understand concepts of 'owner' and 'completely different person' based on the data they have available. Companies can use this robots.txt feature to un-index old marketing content after a re-branding, for example. Or after an acquisition.
Re:Screenshots Forever and Ever Until You Can’t Stand it by StavrosK on Wednesday August 20, 2014 @12:55PM (#8203159)
Sure, but, surely, the bot has timestamps saying "robots.txt allowed me to keep these documents last time I spidered them". Why do they have to be retroactively removed? robots.txt only disallows spidering, it doesn't mandate that you should delete all the data you've already spidered.
Re:Screenshots Forever and Ever Until You Can’t Stand it by db48x on Thursday August 21, 2014 @01:31AM (#8206030)
Because most of the problems come from people who want to hide old material that they didn't realize was being indexed. The automatic behavior is simple and easy to implement, and doesn't require any human intervention.
Re:Screenshots Forever and Ever Until You Can’t Stand it by im3w1l on Wednesday August 20, 2014 @08:16AM (#8201997)
Imo, a better approach than nuking old data, could be to keep the data, but not show it.
Re:Screenshots Forever and Ever Until You Can’t Stand it by psykovsky on Wednesday August 20, 2014 @08:55AM (#8202115)
I may be mistaken, but I think I read somewhere that's exactly what they do. But don't take my word for it.
Re:Screenshots Forever and Ever Until You Can’t Stand it by ersii on Wednesday August 20, 2014 @12:47PM (#8203116)
You are not mistaken. The Internet Archive does not delete or "nuke" the data that is blocked by a robots.txt. Even though cough some people believe so (see parent thread).

Source: IA staffers.

Re:Screenshots Forever and Ever Until You Can’t Stand it by pimlottc on Wednesday August 20, 2014 @02:37AM (#8201318)
What you say is true for sites mirrored by IA's Wayback Machine, though my understanding is they retain the data in case the robots.txt is lifted later on.

Linking to media uploaded to the main archive itself should be safer, though.

Re:Screenshots Forever and Ever Until You Can’t Stand it by walterbell on Wednesday August 20, 2014 @02:33AM (#8201311)
Have you heard of other caches/archives (e.g. Google) applying the same retroactive policy? Presmably IA has no way of finding out that domain ownership has changed. I wonder if they are applying this policy to pages referenced by Wikipedia, http://blog.archive.org/2013/10/25/fixing-broken-links/

The safest archive of a web page is a local PDF.

Re:Screenshots Forever and Ever Until You Can’t Stand it by toomuchtodo on Wednesday August 20, 2014 @03:25AM (#8201398)
> The safest archive of a web page is a local PDF.

Minor quibble: The safest archive of a web page is a local WARC archive:

http://www.archiveteam.org/index.php?title=Wget_with_WARC_ou...

Re:Screenshots Forever and Ever Until You Can’t Stand it by walterbell on Wednesday August 20, 2014 @03:29AM (#8201405)
Good point, I would say both are needed. The WARC is only useful if there is a matching web browser and operating system in a VM which could render the HTML+Javascript and produce the original layout. PDF/A would mostly retain the browser layout.
Re:Screenshots Forever and Ever Until You Can’t Stand it by im3w1l on Wednesday August 20, 2014 @08:23AM (#8202014)
I use virtual notary for this. You give it a URL, and vn fetches it, and gives you a cryptographic certificate of time of retrieval and website's content at that time.
Re:Screenshots Forever and Ever Until You Can’t Stand it by pantaril on Wednesday August 20, 2014 @08:02AM (#8201965)
>I've been thinking a lot the past few weeks about the preservation of art and goods in general.

I've been thinking about this myself. What i found out is that current copyright law is terrible for preservation of our culture because it makes attempts to archive, share, backup or digitalize abandoned games, music or movies illegal before their copyright expires, at which time, most of them would be probably lost foreever.

Re:Screenshots Forever and Ever Until You Can’t Stand it by R_Edward on Wednesday August 20, 2014 @10:29AM (#8202407)
Do you mean to tell me that those crates of my dad's 78 rpm records sitting in my basement might actually be worth something? I was thinking about playing them on a 33.3 rpm turntable into my PC, then using Audacity to speed them up to 78, just to be able to hear the music again, and by association, remember my dad. Geeze, there must be at least a hundred of 'em down there, at least!
Re:Screenshots Forever and Ever Until You Can’t Stand it by randomdestructn on Wednesday August 20, 2014 @01:11PM (#8203234)
Please find proper equipment to play them.

Using a microgroove stylus designed for vinyl will not do your shellac records any favours. Also the recordings will sound terrible.

There are several modern cartridges that support 78rpm stylii, and lots of vintage kit available cheap.

Also careful, they shatter somewhat easily, and like to get mouldy.

Re:Screenshots Forever and Ever Until You Can’t Stand it by keithpeter on Wednesday August 20, 2014 @11:25AM (#8202712)
Google the work of philip jeck (samples on youtube) for hugely emotional use of the old sounds.

Try and get hold of an actual 78 rpm turntable, the old dansettes could do those speeds as could the cheaper separate turntables that had ceramic cartridges with two 'sides' (78/33).

Enjoy

Re:Screenshots Forever and Ever Until You Can’t Stand it by arethuza on Wednesday August 20, 2014 @11:39AM (#8202769)
Rega make a dedicated 78rpm turntable:

http://www.rega.co.uk/rp78.html

Re:Screenshots Forever and Ever Until You Can’t Stand it by lozf on Wednesday August 20, 2014 @11:17AM (#8202669)
If you do this, please, for the sake of any other vinyl you have, use a different (cheaper) stylus for the 78s.
Re:Screenshots Forever and Ever Until You Can’t Stand it by jacquesm on Wednesday August 20, 2014 @05:30AM (#8201653)
I have a few one-off remaining records and I'm very conscious of the responsibility that goes with that. I've never played them and the only time they will be played is when there is a very high grade digitization done at the same time. I've been looking at ways to digitize them without having to actually mechanically play them (using lasers) but those systems are expensive! (http://www.elpj.com/product/price.php)
Re:Screenshots Forever and Ever Until You Can’t Stand it by jerf on Wednesday August 20, 2014 @10:51AM (#8202502)
Well, sure, buying one is expensive, but you're just looking to use one. You only need one to share amongst whomever else you can find who has this need, and the Internet is really good at bringing those sorts of people together. You could Kickstart your way up to owning one of these, just as one idea. (Or work with somebody else to take it on, etc. etc., whatever.)
Re:Screenshots Forever and Ever Until You Can’t Stand it by jacquesm on Wednesday August 20, 2014 @11:28AM (#8202721)
Kickstarter is an interesting option, never thought of that.

You could jumpstart a service that digitizes rare recordings as a service and make them available. Likely there'd be all sorts of copyright issues with a service like that but the basic principle is useful.

Re:Screenshots Forever and Ever Until You Can’t Stand it by jerf on Wednesday August 20, 2014 @12:17PM (#8202957)
Given the sort of thing we're talking about, I think a "ask for forgiveness rather than permission" approach is at least morally justifiable.

I have to admit that before I'd start doing this at scale, I would want some sort of LLC-esque shield in place.

Re:Screenshots Forever and Ever Until You Can’t Stand it by mcb3k on Wednesday August 20, 2014 @10:09AM (#8202340)
Have you looked to see about renting the equipment or finding a service that will help you do this?
Re:Screenshots Forever and Ever Until You Can’t Stand it by jsankey on Wednesday August 20, 2014 @03:45AM (#8201440)
> as my bookmark folders for recipes (RIP BroEats.com) and designs and other things is filled with more and more duds

If you'll forgive me the self-promoting diversion, this is one of the reasons I created my recipe manager app Zest (http://plentyofzest.com/zestapp/). It's frustrating to have a recipe move/disappear, so now I can collect them all in one place.

Re:Screenshots Forever and Ever Until You Can’t Stand it by wlesieutre on Wednesday August 20, 2014 @03:51AM (#8201444)
Are they exportable for 15 years from now when Zest disappears?
Re:Screenshots Forever and Ever Until You Can’t Stand it by jsankey on Wednesday August 20, 2014 @03:59AM (#8201456)
Excellent question and this is on our roadmap (yes, I know, vapourware until we ship it). The plan is to export as HTML with standard microdata annotations (the same we use to automatically import). That should be fairly durable, as far as formats go, and even if the HTML dies at least it is structured (for conversion to a new format) and text-based (at worst you can still read the source).
Re:Screenshots Forever and Ever Until You Can’t Stand it by walterbell on Wednesday August 20, 2014 @02:35AM (#8201315)
A good source of archival resources is http://fileformat.info