Re:Wayback Machine: Now with 240,000,000,000 URLs
by jlgreco on Thursday January 10, 2013 @05:23PM
(#5040170)
I only have second hand insight into how they do it, but my impression is that their spiders log the raw data right to disk and then, periodically (a few times a year?) they process all the accumulated data in one mind-bogglingly massive job. I don't know what the data goes in to as a result of this processing though.
[ Parent
| on HN ]
There are no comments on this comment yet.