Re:Meet PastPages.org, the news homepage archive. (And help ...
by donohoe on Tuesday May 08, 2012 @01:30PM
(#3944809)
Yes.
NYT homepage editor has flexibility on when they "roll" the date depending on news events (and content can be ranked ahead of time too). But yeah, basically midnight.
Internally I know editorial preserves them on an hourly basis. You might ask current NYT-ers if you could get access to that?
I know I did a scrape every 10 (or 2?) minutes of the Homepage HTML for a couple of years (2007 to 2011?). If I can find that data and its still meaningful I'll get it to you. It was quite a few GB.
NYT homepage editor has flexibility on when they "roll" the date depending on news events (and content can be ranked ahead of time too). But yeah, basically midnight.
Internally I know editorial preserves them on an hourly basis. You might ask current NYT-ers if you could get access to that?
I know I did a scrape every 10 (or 2?) minutes of the Homepage HTML for a couple of years (2007 to 2011?). If I can find that data and its still meaningful I'll get it to you. It was quite a few GB.