News for hackers. Stuff that matters.

Meet PastPages.org, the news homepage archive. (And help keep it alive)

http://www.pastpages.org/

I cre­ated this site be­cause I think it ought to ex­ist. The shift­ing homepages of ma­jor me­dia sites should be saved so they can be stud­ied. Done right, I be­lieve Pas­t­Pages could serve as a re­source for schol­ars seek­ing to study cov­er­age of news events, like the up­com­ing U.S. pres­id­en­tial elec­tion.

Regularly collecting the data costs money. So I've organized a Kickstarter in hopes of raising funds to keep it up. http://www.kickstarter.com/projects/651552740/keep-pastpages-alive

( 20 comments | 27 points | on HN )
Re:Meet PastPages.org, the news homepage archive. (And help ... by showerst on Tuesday May 08, 2012 @11:14AM (#3943990)
Do you have permission to be taking these snapshots? The Newseum does something similar for print content and has agreements with all of the organizations so that they don't just Cease & Desist them out of existence.

http://www.newseum.org/todaysfrontpages/

Re:Meet PastPages.org, the news homepage archive. (And help ... by kgen on Tuesday May 08, 2012 @01:54PM (#3944958)
Doesn't Fair Use cover "commentary, criticism, news reporting, research, teaching, library archiving and scholarship."? This site seems non-profit and educational in nature, so I would imagine it would pass the balancing test if push came to shove.
Re:Meet PastPages.org, the news homepage archive. (And help ... by palewire on Tuesday May 08, 2012 @11:02AM (#3943920)
I cre­ated this site be­cause I think it ought to ex­ist. The shift­ing homepages of ma­jor me­dia sites should be saved so they can be stud­ied. Done right, I be­lieve Pas­t­Pages could serve as a re­source for schol­ars seek­ing to study cov­er­age of news events, like the up­com­ing U.S. pres­id­en­tial elec­tion.

Collecting this data cost money. So I've set up a Kickstarter drive to raise funds. If you'd like to help keep PastPages alive, please considering giving.

http://www.kickstarter.com/projects/651552740/keep-pastpages...

Re:Meet PastPages.org, the news homepage archive. (And help ... by atlbeer on Tuesday May 08, 2012 @12:18PM (#3944372)
What stack are you using for the web page capture? It's a perfect crisp capture. I've tried before and never got such good programatic results.
Re:Meet PastPages.org, the news homepage archive. (And help ... by palewire on Tuesday May 08, 2012 @01:09PM (#3944654)
I'm using Selenium's Firefox driver from inside a Django app. There is Python binding that's slick once you figure out a couple timeout related workarounds that are necessary. Their forums helped me over that hurdle.
Re:Meet PastPages.org, the news homepage archive. (And help ... by there on Tuesday May 08, 2012 @11:29AM (#3944097)
I love the New York Times shots. It's a great demonstration of how off-putting their interstitial ads are, and how many other sites don't need them.
Re:Meet PastPages.org, the news homepage archive. (And help ... by palewire on Tuesday May 08, 2012 @11:53AM (#3944240)
Ha! If you know a way around them I'd appreciate the tip.
Re:Meet PastPages.org, the news homepage archive. (And help ... by donohoe on Tuesday May 08, 2012 @01:11PM (#3944669)
Hmm. Not sure if it would make a difference but maybe:

(1) Try this URL instead: http://www.nytimes.com/pages/

(2) Hit the URL by date: http://www.nytimes.com/indexes/yyyy/mm/dd/

Example: http://www.nytimes.com/indexes/2011/12/03/

You can also use this URL structure to get the Homepage back several years (2001) as it was around midnight of that date.

http://www.nytimes.com/indexes/2001/01/01/

http://www.nytimes.com/indexes/2001/09/11/ (Notable)

Not sure if this (or any other section) is of interest too:

http://www.nytimes.com/indexes/2010/12/03/todayspaper/

(3) When you scrape the page find out the link it provides to the Homepage and then try that. I had some success doing that.

What I really want is THIS:

"Reward - NYTimes Login Script" http://donohoe.tumblr.com/post/10723388191/reward-nytimes-lo...

which would get around that problem.

Re:Meet PastPages.org, the news homepage archive. (And help ... by palewire on Tuesday May 08, 2012 @01:18PM (#3944721)
Thanks for this great information. I'm stuck at the jury duty cattle call this morning but will try to put this into action later.
Re:Meet PastPages.org, the news homepage archive. (And help ... by jashkenas on Tuesday May 08, 2012 @01:36PM (#3944835)
Take care with putting this into action -- the external homepage archives constantly suck in the latest version of any precoded module on the page ... so for example, this should show the Iowa Caucuses, but shows nearly two months later instead:

http://www.nytimes.com/indexes/2012/01/03/

... a misstep we later fixed. For what it's worth, there's also an internal version of the homepage archive that doesn't suffer from this problem, and is snapshotted hourly.

Re:Meet PastPages.org, the news homepage archive. (And help ... by danso on Tuesday May 08, 2012 @02:02PM (#3945007)
So even all the external stylesheets/js are (relatively) preserved? That's pretty slick...was this something that was retroactively applied, or something that's existed since early iterations of the CMS?
Re:Meet PastPages.org, the news homepage archive. (And help ... by schwanksta on Tuesday May 08, 2012 @01:15PM (#3944700)
Woah, the /indexes/ url just blew my mind; so it just gives you the last update on that date?
Re:Meet PastPages.org, the news homepage archive. (And help ... by donohoe on Tuesday May 08, 2012 @01:30PM (#3944809)
Yes.

NYT homepage editor has flexibility on when they "roll" the date depending on news events (and content can be ranked ahead of time too). But yeah, basically midnight.

Internally I know editorial preserves them on an hourly basis. You might ask current NYT-ers if you could get access to that?

I know I did a scrape every 10 (or 2?) minutes of the Homepage HTML for a couple of years (2007 to 2011?). If I can find that data and its still meaningful I'll get it to you. It was quite a few GB.

Re:Meet PastPages.org, the news homepage archive. (And help ... by knowtheory on Tuesday May 08, 2012 @12:44PM (#3944510)
I'd think that if you registered a user and submit the session w/ the user cookie you'd probably never hit the interstitial
Re:Meet PastPages.org, the news homepage archive. (And help ... by pavel_lishin on Tuesday May 08, 2012 @12:17PM (#3944364)
How are you fetching them now?
Re:Meet PastPages.org, the news homepage archive. (And help ... by palewire on Tuesday May 08, 2012 @01:11PM (#3944670)
Just a stoopid straight call with Selenium's Python bindings.
Re:Meet PastPages.org, the news homepage archive. (And help ... by adventureful on Tuesday May 08, 2012 @02:28PM (#3945155)
It seems like the NYT doesn't show the interstitial ads on immediate repeat visits (once per day perhaps?). Hit the nyt.com site, then hit it again shortly thereafter and snap that visit.

Can your approach accept cookies?

Re:Meet PastPages.org, the news homepage archive. (And help ... by sp332 on Tuesday May 08, 2012 @12:25PM (#3944413)
I think hosting a static image is cool, but doesn't the Internet Archive already have a full-HTML archive of these pages? e.g. http://web.archive.org/web/20110729013424/http://www.nytimes...
Re:Meet PastPages.org, the news homepage archive. (And help ... by palewire on Tuesday May 08, 2012 @01:07PM (#3944636)
It does an I love that site but unless I'm mistaken I don't think they grab often enough to track fast moving news events.
Re:Meet PastPages.org, the news homepage archive. (And help ... by xabi on Tuesday May 08, 2012 @11:48AM (#3944210)
Same service here, but for newspapers (with more than 1000 newspapers around the globe):

http://en.kiosko.net/us/