News for hackers. Stuff that matters.
Re:Wayback Machine: Now with 240,000,000,000 URLs by ddorian43 on Thursday January 10, 2013 @02:47PM (#5039289)
Does anyone know what database they use? Or just files and folders?
Re:Wayback Machine: Now with 240,000,000,000 URLs by ato on Thursday January 10, 2013 @07:28PM (#5040653)
The index for Wayback is a massive sorted text file (called a CDX) containing a line for each URL and timestamp. For very large installations this index is sharded across multiple servers and queried in parallel. The lookups are done using plain old binary search.

http://archive.org/web/researcher/cdx_file_format.php

Each CDX record maps a URL-timestamp pair to a byte offset into an ARC or WARC file. These are essentially just gzipped HTTP responses concatenated together:

http://archive.org/web/researcher/ArcFileFormat.php http://www.digitalpreservation.gov/formats/fdd/fdd000236.sht...

The document is retrieved, uncompressed, URLs are rewritten, the navigation banner javascript injected and the result is sent to the client.

The code is here: https://github.com/internetarchive/wayback

Re:Wayback Machine: Now with 240,000,000,000 URLs by agatto2 on Saturday January 12, 2013 @04:28PM (#5048958)
How do you get a hold of the list of urls?
Re:Wayback Machine: Now with 240,000,000,000 URLs by jlgreco on Thursday January 10, 2013 @05:23PM (#5040170)
I only have second hand insight into how they do it, but my impression is that their spiders log the raw data right to disk and then, periodically (a few times a year?) they process all the accumulated data in one mind-bogglingly massive job. I don't know what the data goes in to as a result of this processing though.