News for hackers. Stuff that matters.
Re:10,000,000,000,000,000 bytes archived by ChuckMcM on Friday October 26, 2012 @03:56PM (#4703962)
The internet archive is a wonderful thing. I recovered much of my web site when my server burned in a fire, it was cheaper than the $2400 to try to pull it off the melted hard drives. It has also provided fodder for a ton of lawsuits, of the patent/IP/he-said vs she-said varieties.

Given the latter use, and subsequent 'retro-takedowns' that have occurred on the archive, I wonder if there is a market for 'a copy of the archive right now' which would be hard to retro-actively modify? And I wonder what the legal theory would be around having a tape archive of something that was 'clear' at the time you took it, but then later 'redacted'. Could you use your copy of the unredacted information?

Re:10,000,000,000,000,000 bytes archived by JoshTriplett on Friday October 26, 2012 @04:14PM (#4704034)
I wonder if archive.org actually deletes things when taken down, or just makes them inaccessible? Likewise for when archive.org takes content down due to a new robots.txt file that didn't exist on the original site, as often happens with domain squatters.
Re:10,000,000,000,000,000 bytes archived by wtallis on Friday October 26, 2012 @10:32PM (#4705128)
They only disable access, not delete. There was once a court case where the defendant got the court to compel the plaintiff to alter their robots.txt in order to allow the defendant to gather evidence from the archive. (Apparently, the Archive managed to convince the court that manually producing the info hidden in the archive would be too onerous.)
Re:10,000,000,000,000,000 bytes archived by pronoiac on Friday October 26, 2012 @08:55PM (#4704929)
I think they make items "dark" for, say, 70 years, so it's inaccessible to the public. The thought of actual deletions is kind of alien to their goals.
Re:10,000,000,000,000,000 bytes archived by Zenst on Friday October 26, 2012 @03:46PM (#4703912)
I do wonder what the best form of compression would be and given there web pages if some form of customed compression that was optimised for HTML would be useful.

There again with that volume of data, what would CERN do for storage/access for a data pool that size and still be useable.

Reason being if you wanted to back that lot of up ship copies for research purposes then with todays technology the humble memory stick, even the biggest would fail to even handle the file index on this scale. Scary amount of data. But certainly a data set many would like to play with and try things out, being the geeks we are.

Re:10,000,000,000,000,000 bytes archived by juiceandjuice on Friday October 26, 2012 @04:20PM (#4704069)
CERN has a variety of techniques to access, store, and backup their data. I know from experience some experiments, or at least some parts of experiments, use layers and layers of abstraction, like scalla/xrootd which operates on clusters of servers directly or with nfs.

In addition to this, there's levels of processed data. For example, raw data is usually level 0, and basic processed data is level 1 Processed and calibrated data is usually level 2, etc... but experiments often have different definitions for each level. Reprocessing of any later can happen at any time, although level 1 reprocessing is usually an extremely intensive operation because it operates with the largest amount of data.

Level 0 data is usually heavily compressed when it's left on disk, because it's typically the largest amount of data, but also least touched.

Most scientists will use level 2 or level 1 data. This data will be on low latency clusters.

So, while CERN has petabytes of data, typically a fraction of that will be accessible.

In the past, level 0 data was often left to tape. While the raw data is still backed up to tape (I know this is the case for ATLAS), for many experiments with large amounts of data they might leave it on lower cost HDD in simple RAID arrays for redundancy and not worry about performance so much. The BaBar experiment has done this for their long term data analysis.

In addition to all of this, it's still occasionally easier to transfer large amounts of data via tape instead of the internet.

Re:10,000,000,000,000,000 bytes archived by mmagin on Saturday October 27, 2012 @12:40PM (#4706513)
Back when I was involved, the compression was gzip -- each archived page (HTTP headers + body) was separately gzipped and concatenated onto a ~100M file, though I believe they're now doing 1G files. (This cleverly allows retrieving the individual document by seeking to that offset and doing gzip decompression.)

They have significant amounts of inline images and other files in the web archive, which of course typically are already compressed.

http://en.wikipedia.org/wiki/Web_ARChive

Re:10,000,000,000,000,000 bytes archived by taf2 on Friday October 26, 2012 @03:53PM (#4703944)
google does something interesting when you use google.com to convert that number to say gigabytes by doing a search such as: 10,000,000,000,000,000 bytes to gigabytes

the result is 9.31323e6

Notice the e... because it's in the same font your eye might miss it like mine and then you'd say to yourself... 10 gigabytes is so small who cares... but if you do the same search again but this time to petabytes you'll realize it was an 'e' in the gigabyte number...

So google says 8.88178 petabytes that's a lot

Re:10,000,000,000,000,000 bytes archived by rm999 on Friday October 26, 2012 @04:13PM (#4704032)
It's exactly 10 petabytes. Giga/peta are SI prefixes and hence are in base 10.

BTW, take google's conversions with a grain of salt. As I understand it, google's conversion was just a minor project and not something that is maintained at high quality. For these things Wolfam Alpha is almost always better because it's the kind of thing Alpha was designed to do.

Re:10,000,000,000,000,000 bytes archived by cynwoody on Friday October 26, 2012 @10:32PM (#4705131)
Google Calc is a bit behind the curve. The proper prefixes for 2^30 and 2^50 are gibi and pebi, not, as you point out, giga and peta.

http://en.wikipedia.org/wiki/Timeline_of_binary_prefixes

Re:10,000,000,000,000,000 bytes archived by devcpp on Saturday October 27, 2012 @02:58PM (#4706887)
Exactly what he was pointing out.
Re:10,000,000,000,000,000 bytes archived by HyprMusic on Friday October 26, 2012 @04:14PM (#4704036)
The 'e' represents times 10 to the power of the following number. It's a variant of standard form, here's what Wikipedia says about it: http://en.wikipedia.org/wiki/Scientific_notation#E_notation
Re:10,000,000,000,000,000 bytes archived by TazeTSchnitzel on Friday October 26, 2012 @04:29PM (#4704130)
One I hate, too, so easy to miss. *10^ or x10^ is better.
Re:10,000,000,000,000,000 bytes archived by ville on Friday October 26, 2012 @06:10PM (#4704499)
Looks like Google mixes petabytes with pebibytes.
Re:10,000,000,000,000,000 bytes archived by paulsutter on Saturday October 27, 2012 @09:21AM (#4706074)
I love the Internet archive, but is there a typo? At Quantcast we collect more than 10PB of logfiles each year, and we process 30PB of data in a day.

Something seems off about the number

Re:10,000,000,000,000,000 bytes archived by pjscott on Friday October 26, 2012 @08:11PM (#4704828)
That number has a lot of zeros at the end, but really, it's just barely getting started. This may be the greatest electronic civilization archival project in human history, but it is also the smallest and most impoverished. It has a lot of growing left to do.
Re:10,000,000,000,000,000 bytes archived by charonn0 on Friday October 26, 2012 @07:14PM (#4704687)
You know it's a great party if the music is performed live by the guy who wrote the book in your field.
Re:10,000,000,000,000,000 bytes archived by tripzilch on Friday October 26, 2012 @04:15PM (#4704044)
> On Thursday, 25 October, hundreds of Internet Archive supporters, volunteers, and staff celebrated addition of the 10,000,000,000,000,000th byte to the Archive’s massive collections.

So, I bet everyone is dying to know, what was the 10,000,000,000,000,000th byte then?

Re:10,000,000,000,000,000 bytes archived by beernutz on Friday October 26, 2012 @06:52PM (#4704631)
If you look at the picture, they SHOW the actual byte in binary!

looks like it is 236 or 11101100

Re:10,000,000,000,000,000 bytes archived by tankbot on Friday October 26, 2012 @07:30PM (#4704719)
I want to know what this byte was a piece of. Probably Philosoraptor or some cat video.
Re:10,000,000,000,000,000 bytes archived by tripzilch on Friday November 02, 2012 @07:28AM (#4732109)
Shit, I totally missed that, thanks!
Re:10,000,000,000,000,000 bytes archived by TazeTSchnitzel on Friday October 26, 2012 @04:28PM (#4704123)
Considering it may have been binary, and binary would make up the majority of it (images, sounds, video are large), I'm guessing 0x00.
Re:10,000,000,000,000,000 bytes archived by chad_oliver on Friday October 26, 2012 @05:03PM (#4704270)
Assuming a uniform distribution, there's only a 1/256 chance of that.
Re:10,000,000,000,000,000 bytes archived by dkersten on Friday October 26, 2012 @05:14PM (#4704301)
images, sounds, video are large

these are unlikely to be a uniform distribution though

Re:10,000,000,000,000,000 bytes archived by cperciva on Friday October 26, 2012 @05:27PM (#4704344)
If they weren't, they could be compressed better.
Re:10,000,000,000,000,000 bytes archived by espes on Friday October 26, 2012 @05:27PM (#4704342)
These are usually compressed, so likely to be a uniform distribution.
Re:10,000,000,000,000,000 bytes archived by TazeTSchnitzel on Friday October 26, 2012 @05:28PM (#4704350)
The remainder isn't. There are always headers and such.
Re:10,000,000,000,000,000 bytes archived by nealabq on Friday October 26, 2012 @05:08PM (#4704288)
42 of course!

Funny you should asterisk. (42 == '*')

Re:10,000,000,000,000,000 bytes archived by tarice on Friday October 26, 2012 @04:24PM (#4704099)
Interesting note at the bottom for those who may have missed it (Donald Knuth on the organ is very distracting):

>The only thing missing was electricity; the building lost all power just as the presentation was to begin. Thanks to the creativity of the Archive’s engineers and a couple of ridiculously long extension cords that reached a nearby house, the show went on.

Re:10,000,000,000,000,000 bytes archived by 3rd3 on Friday October 26, 2012 @06:40PM (#4704598)
That is already about five times the identifiable storage capacity of the human brain!
Re:10,000,000,000,000,000 bytes archived by desbest on Saturday October 27, 2012 @01:01PM (#4706563)
The Internet Archive uses 10TB more storage every month. I don't know how they do it.
Re:10,000,000,000,000,000 bytes archived by cinch on Saturday October 27, 2012 @01:05PM (#4706577)
i'm interested in their server setup: DIY or vendor?
Re:10,000,000,000,000,000 bytes archived by barbs on Friday October 26, 2012 @07:19PM (#4704700)
Did anyone else read the title as "10 bajillion bytes archived" or was that just me?
Re:10,000,000,000,000,000 bytes archived by arasmussen on Friday October 26, 2012 @03:54PM (#4703948)
"Ten Petabytes (10,000,000,000,000,000 bytes) of cultural material saved!"

Not quite 10 petabytes: (10 * 1024^5) > 10^16

edit: this is wrong, silly me.

Re:10,000,000,000,000,000 bytes archived by anonymouz on Friday October 26, 2012 @04:02PM (#4703987)
I think by now we have more or less standarized on 10^15 as a Petabyte, as this is consistent with the SI system. For the powers-of-2 based approach, the -bi- prefixes are now well established. So:

1 Petabyte = 10^15 Bytes

1 Pebibyte = 1024^5 Bytes = 2^50 Bytes.

See http://en.wikipedia.org/wiki/Petabyte .

This way we no longer need to abuse/redefine the SI prefixes!

edit: definition of Petabyte corrected from 10^16

Re:10,000,000,000,000,000 bytes archived by arasmussen on Friday October 26, 2012 @04:14PM (#4704041)
Oh interesting, thanks for that tidbit of knowledge! I always thought they all (mega, tera, giga, etc) went off the powers-of-2 system.

Also, you have a typo, 1 Petabyte = 10^15 bytes I believe.

Re:10,000,000,000,000,000 bytes archived by anonymouz on Friday October 26, 2012 @04:44PM (#4704191)
> Also, you have a typo, 1 Petabyte = 10^15 bytes I believe.

Thanks!

The power-of-2 system was quite popular for a long time when applied to computers, but always clashed with the much older (and well thought out) SI system where everything is defined in powers of 10, and that is used everywhere else (think physics,...). I think harddrive companies where the first to switch their definitions to the decimal based system (for the obvious reason that their disks would then seem bigger to people accustomed to the common power of 2 system...). A decade or so ago the kibi, mebi, ... prefixes where introduced, and at some point people more or less switched to those. There seems to be an even more detailed article about this at https://en.wikipedia.org/wiki/Binary_prefix .

Re:10,000,000,000,000,000 bytes archived by Aardwolf on Friday October 26, 2012 @04:02PM (#4703988)
Ten petabytes is 10,000,000,000,000,000 bytes. Ten pebibytes is 11,258,999,068,426,240 bytes. So the article is correct.
Re:10,000,000,000,000,000 bytes archived by z92 on Friday October 26, 2012 @07:55PM (#4704790)
10,000,000 GB archived, sounds more cool.