News for hackers. Stuff that matters.
Re:Wayback Machine director outlines the scale of everyone's... by homero on Monday October 08, 2018 @02:04AM (#18164499)
How do they afford all those disks?
Re:Wayback Machine director outlines the scale of everyone's... by mekarpeles on Monday October 08, 2018 @03:56AM (#18165130)
- foundations + government grants https://archive.org/post/2776/who-funds-this

- donations (in-cash / in-kind) https://archive.org/about/credits.php

- digitization services https://archive.org/scanning

- web archiving services https://archive-it.org

src: I work @ Internet Archive on https://openlibrary.org

Re:Wayback Machine director outlines the scale of everyone's... by PM_ME_YOUR_CAT on Monday October 08, 2018 @09:24AM (#18167140)
This is pretty great actually. I see that ads are none of your points: does Wayback Machine either avoid them totally or is it a negligible amount?
Re:Wayback Machine director outlines the scale of everyone's... by aaroninsf on Monday October 08, 2018 @01:36PM (#18169473)
No ads full stop!
Re:Wayback Machine director outlines the scale of everyone's... by Gustomaximus on Monday October 08, 2018 @02:19AM (#18164557)
I've donated token amounts on a few occasions. I feel if you have the means its the right thing to do when a project like this does something useful for you.

I suspect the bulk comes from a few generous individuals / corporations as this is not the most known platform.

Found this with a quick search: https://archive.org/about/credits.php

Re:Wayback Machine director outlines the scale of everyone's... by Dylan16807 on Monday October 08, 2018 @04:25AM (#18165331)
A completely new set of hard drives for 44PB of raw storage would cost 1-1.5 million, and they're only growing by 9% (18%?) per year. It's not all that much, and it's not even close to their biggest budget entry.
Re:Wayback Machine director outlines the scale of everyone's... by GordonS on Monday October 08, 2018 @01:12PM (#18169223)
Don't forget that they will need geo-redundancy built in to prevent data loss, and they likely have some kind of offline archive too.

Then there is the running cost to power all those drives.

Also, when you have that many drives, you're going to be constantly replacing dead ones.

I have a feeling this is an expensive operation!

Re:Wayback Machine director outlines the scale of everyone's... by Dylan16807 on Monday October 08, 2018 @03:14PM (#18170388)
> Don't forget that they will need geo-redundancy built in to prevent data loss

This is already factoring in 2x storage, so you just put each half in a different location.

> and they likely have some kind of offline archive too.

Okay so 2 million.

> Then there is the running cost to power all those drives.

Yes, that's one of many things that adds on top of the raw drive cost.

> Also, when you have that many drives, you're going to be constantly replacing dead ones.

It's not actually that bad! If you cram ten thousand drives into a few racks, your drive failures per week will be in the single digits. You could have someone come in once a month.

> I have a feeling this is an expensive operation!

It is, but my core point is that it would be expensive even if the drives were free. You need a certain amount of server horsepower and the physical resources to support it. This implies that you're not getting to stick 60 drives in a single chassis, so you need more labor as well.

There was an estimate last year of $1500 for IA to store and serve a terabyte of data indefinitely. I'd love to see a breakdown of that number, how much goes to day 1 hardware, how much goes to year 1 hosting, how long that hardware is expected to last. But even the day 1 cost is definitely a lot more than $50 for 2TB worth of hard drive.

Re:Wayback Machine director outlines the scale of everyone's... by GordonS on Monday October 08, 2018 @04:06PM (#18170853)
> This is already factoring in 2x storage, so you just put each half in a different location.

My bad, I missed that!

> It is, but my core point is that it would be expensive even if the drives were free.

On reflection, and after reading your reply, I agree :)

> There was an estimate last year of $1500 for IA to store and serve a terabyte of data indefinitely.

I assume that's $1500 per year?

Re:Wayback Machine director outlines the scale of everyone's... by Dylan16807 on Monday October 08, 2018 @05:25PM (#18171456)
> I assume that's $1500 per year?

Total, not per year. Even S3 would only cost $250 a year to store a TB redundantly, and you could host 20 of them off a $200-per-month server or something.

Backblaze gets you a bit more than one copy for only $60 a year.

Re:Wayback Machine director outlines the scale of everyone's... by GordonS on Tuesday October 09, 2018 @04:00PM (#18179347)
I'm not quite understanding how storing a quantity of data indefinitely doesn't have an ongoing cost?
Re:Wayback Machine director outlines the scale of everyone's... by Dylan16807 on Tuesday October 09, 2018 @08:28PM (#18180966)
Hardware gets cheaper and you can assume a small return on the money, I think. You could also just take it as a guess for 100 years of storage too.