News for hackers. Stuff that matters.
Re:Cloud Storage for $2 per TB per month by kmod on Friday March 06, 2020 @04:44PM (#22507617)
Sure, let's dig into networking. Who pays for rereplication traffic? If you do 64-of-96 RS encoding, that means for every failure you need to transfer 64x the lost storage capacity. If you're targeting a "low individual uptime but high aggregate uptime" model this means you need to be storing data in multiple sites -- and dedicated cross-geo bandwidth is expensive. I agree that in the happy case you can use low-bandwidth cheap equipment, but to get good reliability you need to provision for larger clustered failures such as rack- and row-level outages.
Re:Cloud Storage for $2 per TB per month by Taek on Friday March 06, 2020 @05:33PM (#22507972)
Sure! First up, we don't do repairs every time one host goes down. Standard practice on the network is to wait to do a repair until a full 25% of the redundancy is missing (in 64-of-96, that would be 8 hosts offline). Then you repair all 8 at once, significantly reducing the total amount of repair traffic.

But secondly, offline doesn't usually mean dead and gone, with unstable datacenters like this they are usually back online before the user has lost a full 25% of their redundancy.

Row level and rack level outages are handled by data randomization. The entire Sia system heavily depends on probabilistic techniques, both on the renting and hosting side. Row level failures will take out some of your data, but nobody should be disproportionately impacted by a cluster failure.

On Sia, each piece is at a different site. So 64-of-96 implies that each chunk of data (96 pieces to a chunk) is located in 96 different places. This doesn't help with the geo-bandwidth, but as discussed above there are other techniques to handle that.

Surprisingly, bandwidth pricing on the Sia network is even cheaper than storage pricing relative to centralized competition. That's a lot harder to model at scale though, so we aren't as confident the Sia bandwidth pricing will hold up at $1 / TB in the long term.

And technically, most of this stuff is customizable per-customer. If your particular use case has a different optimal parameterization, it's fairly easy to tune your client to suit your particular needs.

Re:Cloud Storage for $2 per TB per month by jstummbillig on Saturday March 07, 2020 @09:00AM (#22511558)
So, is this it? Dropbox designer challenges the project broadly, gets top comment, Author refutes – and we leave it at that?

I mean this is basically the moment where I would expect every systems designer on HN coming out of the woods and crushing Sia into the ground, if there were, in fact, any ground at all to crush Sia into.

Is this actually legit? If so, where is the rejoicing? What am I missing?

Re:Cloud Storage for $2 per TB per month by zzzcpan on Saturday March 07, 2020 @09:29AM (#22511651)
No, the author's idea is ok and he's mostly right on the cost part. Configuration and the numbers are a bit off and unrealistic, you won't get such low 95% availability per site due to other economical and technological constraints, you'll get at least 99%, but probably closer to three nines per site and 64 out of 96 won't be necessary at all (something like 8 out of 12 could be enough). Dropbox designer is just ignorant, biased and conditioned to US market and environment, but appeals to authority, so people upvote his bad comment. I do storage too, on smaller scale than Dropbox of course, not in the US, but it is distributed and the cost is already lower than what you see in the title.
Re:Cloud Storage for $2 per TB per month by Ajedi32 on Sunday March 08, 2020 @12:02AM (#22516433)
The fact that nobody is able to prove it's a bad idea doesn't necessarily mean it's a good one. There might still be other downsides that haven't been considered, some of which could be solveable with more development work and some not.

At this point the cautious skeptic will be thinking "hmm, maybe there's something to this", not necessarily full on rejoicing.

That said, I agree it does seem promising. If you ever find yourself in need of cheap cloud storage it wouldn't hurt to look into Sia as a possible option.

Re:Cloud Storage for $2 per TB per month by fragmede on Saturday March 07, 2020 @01:32PM (#22513137)
Dropbox designer name dropped "64-of-96 RS encoding" as if they're the only person that's heard of, or dealt with Reed-solomon encoding before, and expected the author to get scared off. There is, in the case of drop box, plenty of ground to crush Sia into. That is the ground between the 95% and multiple-nines of availability.

Engineering is about tradeoffs. I could build a network as good as Google's with infinite money, infinite time, and infinite help. I could design a product as beautify as Apple's with the same lack of limitations. Unfortunately for me, I have limited money, limited time, and limited help. Every systems designer understands that, innately, so isn't rushing out of the woodword because Sia and Dropbox have merely chosen different tradeoffs. That one has IPO'd is uninteresting in the abstract. It's just money after all.

Re:Cloud Storage for $2 per TB per month by tuananh on Monday March 09, 2020 @04:18AM (#22523819)
no. the author mentions "64-of-96" things in the post as well. I don't think kmod means to do what you said.
Re:Cloud Storage for $2 per TB per month by Legogris on Friday March 06, 2020 @05:46PM (#22508084)
That sounds incredibly energy-inefficient. On average you have 12.5% of servers running but not contributing and possibly incurring load on other nodes.
Re:Cloud Storage for $2 per TB per month by H8crilA on Friday March 06, 2020 @10:24PM (#22509525)
12.5% overhead isn't that much. It's what just the networking gear can easily eat in a data center (12% out of all the non-cooling-related power supply).

Reed-Solomon encoding adds 50%, of you want 3 block per 2 data blocks. Replicated encoding (not relevant here since this is allow throughput usecase, but necessary if you want to sustain high read throughput) is adding at least 200% (if you want a 3x replication, which I think should be the minimum).

Re:Cloud Storage for $2 per TB per month by Legogris on Monday March 09, 2020 @06:59AM (#22524392)
12.5% is far from the total overhead, just an additional compounding factor.
Re:Cloud Storage for $2 per TB per month by monocasa on Friday March 06, 2020 @06:02PM (#22508181)
Turn 'em off if they're not contributing.
Re:Cloud Storage for $2 per TB per month by rasz on Friday March 06, 2020 @11:53PM (#22509817)
Spinning iron doesnt like start/stop cycles. Server drives go thru very few in their entire life for that very reason.
Re:Cloud Storage for $2 per TB per month by ec109685 on Saturday March 07, 2020 @12:29PM (#22512769)
These are ssd’s.
Re:Cloud Storage for $2 per TB per month by majdnemkocka on Saturday March 07, 2020 @05:14PM (#22514579)
They are not, see the link in the article: https://www.amazon.com/HGST-7-2K-SATA-Drive-Model/dp/B07XQRB...
Re:Cloud Storage for $2 per TB per month by ec109685 on Saturday March 07, 2020 @11:16PM (#22516313)
Oops. Thanks.
Re:Cloud Storage for $2 per TB per month by dahfizz on Saturday March 07, 2020 @09:14AM (#22511604)
87.5% efficient is not incredibly inefficient.
Re:Cloud Storage for $2 per TB per month by manigandham on Saturday March 07, 2020 @01:04AM (#22510072)
Compared to what though? How efficient is a typical data center? Probably way less than this.
Re:Cloud Storage for $2 per TB per month by DuskStar on Friday March 06, 2020 @05:23PM (#22507913)
I think his point is that he's targeting low aggregate uptime, too.
Re:Cloud Storage for $2 per TB per month by Taek on Friday March 06, 2020 @05:37PM (#22508014)
Definitely not, aggregate uptime is extremely high. We've never seen downtime do to network outages, only software bugs. And even then, only some users were impacted by the bugs, we've never in 5 years had a broad outage.
Re:Cloud Storage for $2 per TB per month by DuskStar on Friday March 06, 2020 @06:22PM (#22508327)
Gotcha, and your reply to the GP clarified a lot for me!