News for hackers. Stuff that matters.
Re:Ice cold archive storage by idlewords on Thursday April 11, 2019 @09:56AM (#19634553)
What is the meaning of the claim about "99.999999999% annual durability"? Does that mean one chance in 100B of an object being unretrievable?
Re:Ice cold archive storage by ajay-d on Thursday April 11, 2019 @10:03AM (#19634627)
Re:Ice cold archive storage by idlewords on Thursday April 11, 2019 @10:08AM (#19634659)
Thank you for that link! It's very interesting to me that all the calculations laid out here assume independent failure events.
Re:Ice cold archive storage by ahelwer on Thursday April 11, 2019 @03:16PM (#19638157)
It is universal practice within cloud service providers to span redundancies across "fault domains" - basically things which could make failures correlated, like being in the same machine/power strip/datacenter/geo region. If you assume your fault domain analysis is good, then your failures should be independent. Many global outages are the result of a previously-unidentified fault domain, like the Azure certificate issues. Of course past a certain point it becomes unimportant - who cares about your data if an asteroid takes out every datacenter on Earth at once?
Re:Ice cold archive storage by gnulinux on Thursday April 11, 2019 @10:16AM (#19634731)
Nothing in engineering is 100%. As far as engineering goes 11 9s is pretty much as good as you can get. For comparison, AWS S3 and Glacier are 11 9s durability too.
Re:Ice cold archive storage by remus on Thursday April 11, 2019 @10:59AM (#19635127)
It's worth bearing in mind the difference between durability and availability. Durability is roughly the chance of losing your data over a given time span (in this case a year), whereas availability is about how reliably you can access the data (and is almost certainly a lot lower than 11 9s). A service can be very durable but have very poor availability.
Re:Ice cold archive storage by rsync on Thursday April 11, 2019 @11:07AM (#19635230)
"What is the meaning of the claim about "99.999999999% annual durability"?"

It has no meaning whatsoever. Someone on the marketing side of the team decided that was a "competitive" number to present, outwards, and someone in engineering was tasked with, working backward from that number, coming up with some plausible calculation that resulted in it.

In the real world, they, like Azure and Amazon, will have single point in time outages that will wipe that out for a year or more.

Here is what an honest assessment looks like:[1]

"Historically (updated April, 2019) we have maintained 99.95% or better availability. It is typical for our storage arrays to have 100+ day uptimes and entire years have passed without interruption to particular offsite storage arrays."

...

"In the event of a conflict between data integrity and uptime, rsync.net will ALWAYS choose data integrity."

[1] https://www.rsync.net/resources/notices/sla.html

Re:Ice cold archive storage by jefftk on Thursday April 11, 2019 @11:23AM (#19635470)
> In the real world, they, like Azure and Amazon, will have single point in time outages that will wipe that out for a year or more.

An outage affects availability, but as long as it's not permanent it doesn't affect durability. For example, if I add a new backup provider that stores data on-premise I've added a (nearly) independent data store. This substantially decreases my risk of losing my data unrecoverably (increases durability) but if I don't set up any sort of automatic failover I'm still at risk for substantial outages (no practical increase in availability).

For example, I don't believe Amazon has ever lost any S3 data (https://www.quora.com/Has-Amazon-S3-ever-lost-data-permanent...), and if they did it would be a big deal. Same with the other major cloud storage providers.

> Someone on the marketing side of the team decided that was a "competitive" number to present, outwards, and someone in engineering was tasked with, working backward from that number, coming up with some plausible calculation that resulted in it.

I would be incredibly surprised if that happened. That's not the way I've seen anyone work here.

(Disclosure: I work at Google, though not in Cloud)

Re:Ice cold archive storage by jpatokal on Thursday April 11, 2019 @11:25AM (#19635501)
You are mixing availability (access at any given moment) with durability (not losing data). From the FAQ:

Cloud Storage is designed for 99.999999999% (11 9's) annual durability, which is appropriate for even primary storage and business-critical applications. This high durability level is achieved through erasure coding that stores data pieces redundantly across multiple devices located in multiple availability zones.

Disclaimer: I work at GCP, although not in GCS specifically.

Re:Ice cold archive storage by rsync on Thursday April 11, 2019 @12:03PM (#19636004)
"You are mixing availability (access at any given moment) with durability (not losing data)."

You are correct - I misread that as availability even after quoting that very same line.