News for hackers. Stuff that matters.
Re:Cloud Storage for $2 per TB per month by kmod on Friday March 06, 2020 @03:43PM (#22507135)
I worked on the design of Dropbox's exabyte-scale storage system, and from that experience I can say that these numbers are all extremely optimistic, even with their "you can do it cheaper if you only target 95% uptime" caveat. Networking is much more expensive, labor is much more expensive, space is much more expensive, depreciation is faster than they say, etc etc. I don't think the authors have ever done any actual hardware provisioning before.

I didn't read all their math but I expect their final result to be off by a factor of 2-5x. Hard drives are a surprisingly low percentage of the cost of a storage system.

Re:Cloud Storage for $2 per TB per month by Taek on Friday March 06, 2020 @04:05PM (#22507281)
Author here. A lot of these numbers are drawn from experience in the mining world, where people realized that when cost is the ultimate bottom line, a lot of corners can be cut.

Sia systems don't need a ton of networking. I ran the networking buildout costs by some networking people, and again it comes down to cutting corners. If you only need 10 gbps per rack, if you don't mind having extra milliseconds added, etc, you can get away with very scrappy setups. The whole point is that it's not a highly reliable facility.

Re:Cloud Storage for $2 per TB per month by kmod on Friday March 06, 2020 @04:44PM (#22507617)
Sure, let's dig into networking. Who pays for rereplication traffic? If you do 64-of-96 RS encoding, that means for every failure you need to transfer 64x the lost storage capacity. If you're targeting a "low individual uptime but high aggregate uptime" model this means you need to be storing data in multiple sites -- and dedicated cross-geo bandwidth is expensive. I agree that in the happy case you can use low-bandwidth cheap equipment, but to get good reliability you need to provision for larger clustered failures such as rack- and row-level outages.
Re:Cloud Storage for $2 per TB per month by Taek on Friday March 06, 2020 @05:33PM (#22507972)
Sure! First up, we don't do repairs every time one host goes down. Standard practice on the network is to wait to do a repair until a full 25% of the redundancy is missing (in 64-of-96, that would be 8 hosts offline). Then you repair all 8 at once, significantly reducing the total amount of repair traffic.

But secondly, offline doesn't usually mean dead and gone, with unstable datacenters like this they are usually back online before the user has lost a full 25% of their redundancy.

Row level and rack level outages are handled by data randomization. The entire Sia system heavily depends on probabilistic techniques, both on the renting and hosting side. Row level failures will take out some of your data, but nobody should be disproportionately impacted by a cluster failure.

On Sia, each piece is at a different site. So 64-of-96 implies that each chunk of data (96 pieces to a chunk) is located in 96 different places. This doesn't help with the geo-bandwidth, but as discussed above there are other techniques to handle that.

Surprisingly, bandwidth pricing on the Sia network is even cheaper than storage pricing relative to centralized competition. That's a lot harder to model at scale though, so we aren't as confident the Sia bandwidth pricing will hold up at $1 / TB in the long term.

And technically, most of this stuff is customizable per-customer. If your particular use case has a different optimal parameterization, it's fairly easy to tune your client to suit your particular needs.

Re:Cloud Storage for $2 per TB per month by jstummbillig on Saturday March 07, 2020 @09:00AM (#22511558)
So, is this it? Dropbox designer challenges the project broadly, gets top comment, Author refutes – and we leave it at that?

I mean this is basically the moment where I would expect every systems designer on HN coming out of the woods and crushing Sia into the ground, if there were, in fact, any ground at all to crush Sia into.

Is this actually legit? If so, where is the rejoicing? What am I missing?

Re:Cloud Storage for $2 per TB per month by zzzcpan on Saturday March 07, 2020 @09:29AM (#22511651)
No, the author's idea is ok and he's mostly right on the cost part. Configuration and the numbers are a bit off and unrealistic, you won't get such low 95% availability per site due to other economical and technological constraints, you'll get at least 99%, but probably closer to three nines per site and 64 out of 96 won't be necessary at all (something like 8 out of 12 could be enough). Dropbox designer is just ignorant, biased and conditioned to US market and environment, but appeals to authority, so people upvote his bad comment. I do storage too, on smaller scale than Dropbox of course, not in the US, but it is distributed and the cost is already lower than what you see in the title.
Re:Cloud Storage for $2 per TB per month by Ajedi32 on Sunday March 08, 2020 @12:02AM (#22516433)
The fact that nobody is able to prove it's a bad idea doesn't necessarily mean it's a good one. There might still be other downsides that haven't been considered, some of which could be solveable with more development work and some not.

At this point the cautious skeptic will be thinking "hmm, maybe there's something to this", not necessarily full on rejoicing.

That said, I agree it does seem promising. If you ever find yourself in need of cheap cloud storage it wouldn't hurt to look into Sia as a possible option.

Re:Cloud Storage for $2 per TB per month by fragmede on Saturday March 07, 2020 @01:32PM (#22513137)
Dropbox designer name dropped "64-of-96 RS encoding" as if they're the only person that's heard of, or dealt with Reed-solomon encoding before, and expected the author to get scared off. There is, in the case of drop box, plenty of ground to crush Sia into. That is the ground between the 95% and multiple-nines of availability.

Engineering is about tradeoffs. I could build a network as good as Google's with infinite money, infinite time, and infinite help. I could design a product as beautify as Apple's with the same lack of limitations. Unfortunately for me, I have limited money, limited time, and limited help. Every systems designer understands that, innately, so isn't rushing out of the woodword because Sia and Dropbox have merely chosen different tradeoffs. That one has IPO'd is uninteresting in the abstract. It's just money after all.

Re:Cloud Storage for $2 per TB per month by tuananh on Monday March 09, 2020 @04:18AM (#22523819)
no. the author mentions "64-of-96" things in the post as well. I don't think kmod means to do what you said.
Re:Cloud Storage for $2 per TB per month by Legogris on Friday March 06, 2020 @05:46PM (#22508084)
That sounds incredibly energy-inefficient. On average you have 12.5% of servers running but not contributing and possibly incurring load on other nodes.
Re:Cloud Storage for $2 per TB per month by H8crilA on Friday March 06, 2020 @10:24PM (#22509525)
12.5% overhead isn't that much. It's what just the networking gear can easily eat in a data center (12% out of all the non-cooling-related power supply).

Reed-Solomon encoding adds 50%, of you want 3 block per 2 data blocks. Replicated encoding (not relevant here since this is allow throughput usecase, but necessary if you want to sustain high read throughput) is adding at least 200% (if you want a 3x replication, which I think should be the minimum).

Re:Cloud Storage for $2 per TB per month by Legogris on Monday March 09, 2020 @06:59AM (#22524392)
12.5% is far from the total overhead, just an additional compounding factor.
Re:Cloud Storage for $2 per TB per month by monocasa on Friday March 06, 2020 @06:02PM (#22508181)
Turn 'em off if they're not contributing.
Re:Cloud Storage for $2 per TB per month by rasz on Friday March 06, 2020 @11:53PM (#22509817)
Spinning iron doesnt like start/stop cycles. Server drives go thru very few in their entire life for that very reason.
Re:Cloud Storage for $2 per TB per month by ec109685 on Saturday March 07, 2020 @12:29PM (#22512769)
These are ssd’s.
Re:Cloud Storage for $2 per TB per month by majdnemkocka on Saturday March 07, 2020 @05:14PM (#22514579)
They are not, see the link in the article: https://www.amazon.com/HGST-7-2K-SATA-Drive-Model/dp/B07XQRB...
Re:Cloud Storage for $2 per TB per month by ec109685 on Saturday March 07, 2020 @11:16PM (#22516313)
Oops. Thanks.
Re:Cloud Storage for $2 per TB per month by dahfizz on Saturday March 07, 2020 @09:14AM (#22511604)
87.5% efficient is not incredibly inefficient.
Re:Cloud Storage for $2 per TB per month by manigandham on Saturday March 07, 2020 @01:04AM (#22510072)
Compared to what though? How efficient is a typical data center? Probably way less than this.
Re:Cloud Storage for $2 per TB per month by DuskStar on Friday March 06, 2020 @05:23PM (#22507913)
I think his point is that he's targeting low aggregate uptime, too.
Re:Cloud Storage for $2 per TB per month by Taek on Friday March 06, 2020 @05:37PM (#22508014)
Definitely not, aggregate uptime is extremely high. We've never seen downtime do to network outages, only software bugs. And even then, only some users were impacted by the bugs, we've never in 5 years had a broad outage.
Re:Cloud Storage for $2 per TB per month by DuskStar on Friday March 06, 2020 @06:22PM (#22508327)
Gotcha, and your reply to the GP clarified a lot for me!
Re:Cloud Storage for $2 per TB per month by arcticbull on Friday March 06, 2020 @04:32PM (#22507511)
Here's the issue. We know that due to economy of scale and domain experience, AWS will always have the lowest cost (to Amazon) for storage -- whether that's totally-reliable storage, or sorta-reliable. If there was a demand for sorta-reliable, they'd build a sorta-reliable S3 and undercut you. Then, blockchain adds inefficiency. Therefore, it's basically impossible for any blockchain solution to have a lower total cost to provide storage.
Re:Cloud Storage for $2 per TB per month by Dylan16807 on Friday March 06, 2020 @08:14PM (#22508989)
> If there was a demand for sorta-reliable, they'd build a sorta-reliable S3 and undercut you.

Amazon's goal is to get people to pay a premium to get access to their entire ecosystem of services. They don't optimize the price of each individual service. There's a lot of take it or leave it for their specific offerings. Look at how they used to have reduced redundancy as a cost saver, but don't anymore. If you want corner-cutting storage from Amazon, you get funneled into Glacier now.

So let's look at how glacier deep archive is $.99 per TB per month, with a waiting time. And Google's weird offering where it costs $1.23 per TB per month, with instant access, but you pay a bunch of money to access. That means they can't store it on tapes, but they expect to profit anyway. And they're probably not running that as a loss leader that depends on the archival data being accessed.

Re:Cloud Storage for $2 per TB per month by manigandham on Saturday March 07, 2020 @01:07AM (#22510084)
S3 Reduced Redundancy is replaced with S3 Infrequent Access (and another lower tier One-Zone Infrequent Access) so there's still some pricing flexibility available.
Re:Cloud Storage for $2 per TB per month by Dylan16807 on Saturday March 07, 2020 @01:20AM (#22510119)
Infrequent Access is closer. But wow, one-zone sure is only 20% cheaper than three-zone. And the retrieval cost is significantly more expensive than glacier's bulk price.

Actually, looking closer, is one-zone even storing fewer copies? It's offering the same durability, except in the case of "availability zone destruction". So if they're selling three copies for $10 a TB, then the equivalent price for 64-of-96 would be $5 a TB. Cutting that down with cheaper worse hardware would go a long way to get you toward the $2 goal.

Re:Cloud Storage for $2 per TB per month by manigandham on Saturday March 07, 2020 @03:27AM (#22510552)
S3 Standard and S3 IA store multiple copies spread across 3 zones. S3 One-Zone IA is stored as multiple copies but all within a single zone. They give you the same technical durability but less availability if that zone goes down, and if it's destroyed then you lose all your data.
Re:Cloud Storage for $2 per TB per month by hinkley on Saturday March 07, 2020 @07:23PM (#22515259)
That has the potential to cut both ways. If Amazon decides that S3 is their loss leader, then they would absolutely bury boutique players who only cover a few functional areas.

If instead they decide it's the cash cow, then it wouldn't be a stretch to predict that they price it a bit above the competition. The customer has the impression that they can amortize the cost across the other value-adds in the ecosystem. Whether the customer is right or not barely matters to Amazon from a purely financial aspect. All that matters is that they believe it to be true.

There's also the short-term versus the long term strategy. Short term they could do either and I wouldn't be surprised. Long term, I think I would expect the latter.

Re:Cloud Storage for $2 per TB per month by NohatCoder on Saturday March 07, 2020 @11:21AM (#22512325)
Where did you get those prices? I can't find anything cheaper than $4 per TB month at both Google and Amazon.
Re:Cloud Storage for $2 per TB per month by Dylan16807 on Saturday March 07, 2020 @01:17PM (#22513045)
https://aws.amazon.com/s3/pricing/

Glacier Deep Archive is .99/TB/month in US East and EU Ireland and several others, plus $2.50/TB bulk retrieval.

https://cloud.google.com/storage/pricing

Archive Storage is $1.20/TB/month in us-east1 and europe-west1 and several others. But (instant) retrieval is $50/TB, with no options...

Re:Cloud Storage for $2 per TB per month by throwaway9878 on Friday March 06, 2020 @04:41PM (#22507601)
Everybody with more money than you can always undercut you in anything you ever do so why bother ever trying to do anything
Re:Cloud Storage for $2 per TB per month by enaaem on Friday March 06, 2020 @06:57PM (#22508538)
Very good point! Let's say you move into a new market, trying to undercut the status quo. The status quo, with their power of scale and experience can just undercut you back. Who is benefitting from this? The customers! So if the customers want more competition they have to pay you to play. Which means they have to co-invest with you and promise to buy your service later.

A good is example are the Apple iPhone screens. If Apple wants a new competing supplier, they have to invest in new competitors.

Re:Cloud Storage for $2 per TB per month by nojvek on Saturday March 07, 2020 @02:43PM (#22513572)
Not really. Suppose giant A undercuts startup B where A is making a temporary loss but B runs out of money trying to complete, only A remains in the market and A starts charging a premium since it doesn’t have completion.

This is bad for customers in the long term. Like how Amazon is slowly destroying brick and mortar stores, or having Amazon Basics at the front undercutting other sellers. Monopolies mean you don’t have an open fair market anymore.

Re:Cloud Storage for $2 per TB per month by enaaem on Sunday March 08, 2020 @07:35AM (#22517533)
It's true that benefits for customers would likely be only temporary. But that is not my main point.

I'm explaining a common business move called pay to play, in industries that require very high investments for new competitors. If customers want B to compete, they have to pay B first to ensure B does not make a loss, so everyone wins a little in the end.

So in the case of OP. He needs to finds paying customers first, who are willing to pre-order his service.

Re:Cloud Storage for $2 per TB per month by nemonemo on Friday March 06, 2020 @04:49PM (#22507659)
I think the point is that the cost should not be the main motivator. I would agree that there needs to be other differentiators in addition to cost, which could provide sufficient moat against other competitors, big or small.
Re:Cloud Storage for $2 per TB per month by jessriedel on Friday March 06, 2020 @05:04PM (#22507768)
It's possible that there could be non-obvious innovations in how to save money with a low reliability threshold, which Amazon might not be able to effortlessly copy.
Re:Cloud Storage for $2 per TB per month by res0nat0r on Friday March 06, 2020 @06:05PM (#22508213)
It should also be noted that you can get S3 storage for $1/TB/month already if you use the Glacier Deep Archive storage class.
Re:Cloud Storage for $2 per TB per month by paulryanrogers on Friday March 06, 2020 @06:26PM (#22508351)
Did that include network IO to retrieve your data?
Re:Cloud Storage for $2 per TB per month by notyourday on Friday March 06, 2020 @08:30PM (#22509075)
Nope. That's what actually kills you on Amazon - network costs.
Re:Cloud Storage for $2 per TB per month by Dylan16807 on Saturday March 07, 2020 @01:47AM (#22510208)
It's... interesting... that Amazon offers reasonably priced bandwidth on Lightsail, but it's against the TOS to use it in connection with other services.
Re:Cloud Storage for $2 per TB per month by notyourday on Saturday March 07, 2020 @07:04AM (#22511180)
Yeah, I have been proxy accessing my personal S3 that way, but I'm not really willing to bet a business on it.
Re:Cloud Storage for $2 per TB per month by zadokshi on Saturday March 07, 2020 @04:12PM (#22514102)
Not really. Have you ever done the maths on what it costs to recover 1TB in a recovery situation.

Definitely not worth it.

Re:Cloud Storage for $2 per TB per month by Dylan16807 on Saturday March 07, 2020 @08:47PM (#22515713)
Putting it into normal S3 only costs $2.50 a TB. That's very affordable.

To get out of Amazon entirely, they definitely want to gouge you, but it's not the end of the world. If $24 a year is an acceptable storage price, then Snowball export costing somewhere around $36/TB in bulk isn't too awful. (And if you don't have enough data to fill up a snowball, you can probably smuggle it out through lightsail.)

Re:Cloud Storage for $2 per TB per month by arcticbull on Friday March 06, 2020 @04:57PM (#22507718)
My point is taking something, adding inefficiency, then pronouncing it's "cheapest in class" isn't logically possible.
Re:Cloud Storage for $2 per TB per month by badestrand on Friday March 06, 2020 @10:15PM (#22509498)
You are confusing "cheapest option possible" with "cheapest option available". Maybe Amazon _could_ be cheapest and most efficient in everything but they aren't, that's why they make a lot of profit with their cloud services. Thus there can easily be a competitor with a cheaper offering, especially if they are cutting some corners as per the article, even if they have some inefficiencies in other areas.
Re:Cloud Storage for $2 per TB per month by paxys on Friday March 06, 2020 @05:24PM (#22507915)
The point is not to do it cheaper but better.
Re:Cloud Storage for $2 per TB per month by wju3h3uwh on Friday March 06, 2020 @06:55PM (#22508526)
Quantity is a type of quality. Lots of people mainly use cloud storage as a cold storage locker for backing up photos and documents. I'm sure there's ways to innovate in the space that would make certain clouds more appealing, but most solutions are already effective as simple storage lockers. Doing it better instead of cheaper at this point is more likely to be either a megacorp affiliation perk, like a discount on a YouTube Music subscription, or phone company style family plan discounts "Get discounted storage for the whole family if you pay slightly less than the price of four memberships!"
Re:Cloud Storage for $2 per TB per month by arcticbull on Friday March 06, 2020 @06:32PM (#22508389)
... better than AWS with its SLAs, that every major company relies on to some extent?
Re:Cloud Storage for $2 per TB per month by paxys on Friday March 06, 2020 @06:48PM (#22508484)
Yes. There's a reason companies like Digital Ocean, Heroku, OVH, ZEIT, Joyent, Rackspace, Linode, Cloudflare and many more have been able to survive and grow rapidly in an AWS-dominated space. None of them are competing by undercutting Amazon in price.
Re:Cloud Storage for $2 per TB per month by folmar on Thursday March 19, 2020 @04:14PM (#22631200)
OVH is certainly undercutting Amazon in price. The free bandwidth included in each instance is already a dealbreaker if you actually use the instance for anything other than very heavy cpu-bound loads with small output to send back.
Re:Cloud Storage for $2 per TB per month by DarthGhandi on Friday March 06, 2020 @08:38PM (#22509120)
Re:Cloud Storage for $2 per TB per month by Dylan16807 on Saturday March 07, 2020 @12:25AM (#22509936)
"Better" in a niche-filling way, not a "this is very well engineered, but for my use case it's very overengineered" way.
Re:Cloud Storage for $2 per TB per month by vincentlee on Friday March 06, 2020 @08:16PM (#22509000)
market your product better, steal better companies people, lie, play political games, leverage relationships, kill

so many options

Re:Cloud Storage for $2 per TB per month by yourapostasy on Friday March 06, 2020 @05:22PM (#22507895)
> ...they'd build a sorta-reliable S3...

From operational experience reports of S3 I've seen out there and discussions [1], including on HN [2], once you reach a large enough object count, S3 is "sorta-reliable". Objects will disappear. The durability claim is not SLA-enforced [3], which is uptime-oriented. The only remedy is a service credit.

[1] https://www.theregister.co.uk/2018/07/19/data_durability_sta...

[2] https://news.ycombinator.com/item?id=14880617

[3] https://aws.amazon.com/s3/sla/

Re:Cloud Storage for $2 per TB per month by dannyw on Saturday March 07, 2020 @04:36AM (#22510801)
Had about 20k objects and lost one (405s no matter what) about six month in.

I didn’t think much about it, but I guess I was really unlucky.

Re:Cloud Storage for $2 per TB per month by kmod on Friday March 06, 2020 @05:12PM (#22507829)
I agree that AWS has economies of scale that makes it hard to build a better/cheaper S3, but one way you can get around this problem is by building to lesser requirements. S3 has to work for all use cases, but if you know you need less of something costly (say, less IOPs) you can build a system that's cheaper than S3 even if the individual components are more expensive than S3 is paying.
Re:Cloud Storage for $2 per TB per month by notyourday on Friday March 06, 2020 @05:28PM (#22507933)
This is actually not correct.

Because of AWS scale it cannot be the cheapest. Amazon cannot use odd lots or small lots, which means that it has very few possible suppliers and those suppliers will never be the cheapest.

Same goes for the network infrastructure. While it is possible buy 100Gbit/sec at a throw away price to because some sales person needs to make his numbers and neither he nor his director of sales thinks Kmod Hosting would be able to fill the pipe it is buying, Amazon would need 900Gbit/sec for their exit and no one at the vendor is going to blink at making Amazon pay higher price than Kmod Hosting.

Re:Cloud Storage for $2 per TB per month by late2part on Friday March 06, 2020 @06:53PM (#22508513)
you are wrong that AWS pays more than small vendors.
Re:Cloud Storage for $2 per TB per month by jjeaff on Saturday March 07, 2020 @01:00AM (#22510058)
That's not what they said. They pay more than small vendors, sometimes.

If you are able to take advantage of sales and clearance deals here and there, you are very likely to get a lower price than even the biggest buyers.

Re:Cloud Storage for $2 per TB per month by dannyw on Saturday March 07, 2020 @04:33AM (#22510792)
1 month of sales hunting has got me a PC rig for literally 2/3rd of the cost of buying without waiting. And this was an optimised build in the first place.
Re:Cloud Storage for $2 per TB per month by vidarh on Saturday March 07, 2020 @01:38AM (#22510181)
They probably could,but they're showing no sign of wanting to compete on cost.

E.g bandwidth costs on AWS are high enough that if you actually serve up lots of data from S3 you can typically afford to rent servers to cache all of it 'in front' of AWS and still save a ton of money.

S3 only gets close to competitive if you never access the data from outside of AWS.

Which gets to the point: If you use an AWS service like S3, you pretty much has to use other AWS services if you want the cost to be even somewhat reasonable in aggregate.

They don't need to compete on cost, because once they get you to buy into one set of services, moving any one set of services off AWS gets more painful and/or costly, and a full migration looks too scary for most people.

S3 will never be priced to undercut anything but big players for that reason. They need to be competitive with Google and Azure, because those guys can offer to offset transitioning costs and generally aggressively target AWS customers.

A small player isn't the same threat even if substantially cheaper.

Re:Cloud Storage for $2 per TB per month by Taek on Friday March 06, 2020 @05:57PM (#22508151)
Sia is a lot more than a low cost storage platform. It's a full reimagining of how the cloud should work with an emphasis on user ownership and control, open access for developers, and ultimately decentralization that allows users to be certain their applications will always work (no more "RSS Reader is shutting down").

This post is intended to address people who do not feel that decentralization can be cost effective at scale.

Re:Cloud Storage for $2 per TB per month by jrochkind1 on Friday March 06, 2020 @06:14PM (#22508272)
Where do I find the introduction what Sia is? I tried clicking around in the headers from OP, but couldn't find it within my very lazy tolerance.
Re:Cloud Storage for $2 per TB per month by Taek on Friday March 06, 2020 @06:29PM (#22508368)
https://blog.sia.tech/skynet-bdf0209d6d34 - this is probably the best thing we have at the moment.

https://support.sia.tech/article/dk91b0eibc-welcome-to-sia - this support article is also a good introduction to the network and why it was built

Re:Cloud Storage for $2 per TB per month by beedrillzzzzz on Friday March 06, 2020 @06:31PM (#22508377)
Website: https://sia.tech/

Also take a look at Skynet, a file sharing protocol built on top of Sia: https://siasky.net/

Re:Cloud Storage for $2 per TB per month by easygenes on Friday March 06, 2020 @07:51PM (#22508866)
I cringe every time a project uses the name "Skynet". First, it's overused. Second, it's literally the name of the AI project that went rogue and tried to kill all the humans in the Terminator movie universe. Not exactly the best association. Just find another name.
Re:Cloud Storage for $2 per TB per month by ethbro on Friday March 06, 2020 @08:45PM (#22509161)
On the other hand, I am filled with positive associations at the name.

Reliability, redundancy, disaster recovery, cost cutting, and absolute focus on execution!

Re:Cloud Storage for $2 per TB per month by Evidlo on Friday March 06, 2020 @08:43PM (#22509151)
See Soylent.
Re:Cloud Storage for $2 per TB per month by ineedasername on Friday March 06, 2020 @09:37PM (#22509344)
What aspect of this infrastructure is user-owned?
Re:Cloud Storage for $2 per TB per month by KarlKemp on Saturday March 07, 2020 @01:36AM (#22510176)
You re certainly reimagining the marketing slang of a few years ago (a decade for the gratuitous dig at Google Reader)
Re:Cloud Storage for $2 per TB per month by risyachka on Saturday March 07, 2020 @02:58AM (#22510461)
Amazon has huge margins. They could sell you the same services for waaaaaay less money. They just won't.
Re:Cloud Storage for $2 per TB per month by jknoepfler on Friday March 06, 2020 @04:48PM (#22507648)
having worked at aws, all I'll say is I think you're wildly over-estimating their agility
Re:Cloud Storage for $2 per TB per month by Hamuko on Friday March 06, 2020 @04:59PM (#22507735)
Care to share any funny stories?
Re:Cloud Storage for $2 per TB per month by scarface74 on Friday March 06, 2020 @05:33PM (#22507976)
Isn’t that what AWS S3 one zone infrequent access is?
Re:Cloud Storage for $2 per TB per month by animalnewbie on Friday March 06, 2020 @10:43PM (#22509599)
> (paraphrased) domain experience means they ae best

As someone who has worked with a couple of market leaders in their respective fields i want to disssaude anyone who would listen of this notion. NO, the market leaders aren't doing things optimally. Some things are downright stupid.

Re:Cloud Storage for $2 per TB per month by kllrnohj on Friday March 06, 2020 @08:08PM (#22508953)
The buildout in the article doesn't work. You can't plug in that 4-lane SAS SFF-8087 splitter cable into that motherboard. You're only getting 8 hard drives per motherboard with that setup, not 32.

That puts the cost of 192 TB at more like $6240‬, not $4945. Could be less if you find a good deal on mini-SAS PCIE cards, but still going to be substantially higher than $4945.

Re:Cloud Storage for $2 per TB per month by ADefenestrator on Saturday March 14, 2020 @07:14PM (#22579511)
Even if it had slots for the splitter cable, Intel and AMD onboard SATA explicitly doesn't support port multipliers as far as I know. You can buy PCIE SAS cards that do for relatively cheap, but then you have to find board with enough PCIe slots. Easy enough on the "gamer" boards but if you want ECC (and you probably do, for storage) and IPMI (you probably do, if you have more than a few dozen servers) your options get much more limited. Other than 1 or 2 Asrock Rack boards, you pretty much have to move into Epyc 7000-series or Xeon Silver or above. Often dual-socket on the Xeons to get a board with lots of PCIe.

In theory something like an Epyc 3000-series with lots of PCIe or onboard SATA that supports port multipliers would work great, but I don't think anyone actually makes that.

Re:Cloud Storage for $2 per TB per month by notyourday on Friday March 06, 2020 @08:29PM (#22509072)
You are going to get a 4U server that takes 60 disks and populate it with SAS cards driving them using SAS-2-SATA cables.
Re:Cloud Storage for $2 per TB per month by anonymousiam on Friday March 06, 2020 @04:21PM (#22507426)
The third sentence of your Medium article says; "Despite this, the Sia network is able to achieve 99.9999% uptime for files." How do you achieve this in a "not a highly reliable facility"?
Re:Cloud Storage for $2 per TB per month by prepend on Friday March 06, 2020 @04:23PM (#22507443)
Redundancy, I assume. Lots of unreliable facilities and nodes can be very durable in aggregate.
Re:Cloud Storage for $2 per TB per month by jakewalker on Friday March 06, 2020 @04:54PM (#22507694)
Sounds kinda like the people who thought that bundling a bunch of bad mortgage debt together in slices could get it rated AAA.
Re:Cloud Storage for $2 per TB per month by rpdillon on Friday March 06, 2020 @08:14PM (#22508988)
It does sound like that! The mistake there was to assume the failure of one mortgage was statistically independent of the failure of another, which is obviously incorrect for many failure scenarios. In this case, it it would be similar if all nodes were in, say, the same datacenter. That doesn't appear to be the case, but there may be other dimensions on which the network lacks the required diversity to support the reliability claims (disk vendor and age...? There must be others.)
Re:Cloud Storage for $2 per TB per month by feross on Friday March 06, 2020 @05:43PM (#22508065)
Your mind is going to be blown when you learn how TCP/IP works.
Re:Cloud Storage for $2 per TB per month by allendoerfer on Sunday March 08, 2020 @03:04PM (#22519856)
Or computers for that matter. Deep down it is not really about 1s and 0s, more like thresholds in between 1 and 0.

To me a big part of computer science is abstracting away unreliable details to make them seem reliable.

Re:Cloud Storage for $2 per TB per month by justinmeiners on Friday March 06, 2020 @08:45PM (#22509158)
Not really. Just probability. If you have fully redundant services then ALL of them have to go down to have an outage. Suppose you 5 copies with 75% uptime each. The probability that all of them are down is 0.25^5 ~ 0.0009

Now of course that assumes they are uncorrelated, but since Sia nodes are distributed across the internet, that's likely as opposed to multiple servers at a few data centers like AWS.

Turns out mortgage bonds tend to be correlated.

Re:Cloud Storage for $2 per TB per month by iampims on Friday March 06, 2020 @09:16PM (#22509271)
They all run the same software stack though, so despite being deployed on diverse hardware, so they can’t claim to only have independent failure modes.
Re:Cloud Storage for $2 per TB per month by justinmeiners on Friday March 06, 2020 @11:27PM (#22509728)
True. It would be interesting to analyze what others aspects are correlated.
Re:Cloud Storage for $2 per TB per month by __jal on Friday March 06, 2020 @05:18PM (#22507861)
Reminds me of a saying from the first dot.com crash. (Or at least I heard it then first.)

Tying two bricks together doesn't make them float.

Re:Cloud Storage for $2 per TB per month by kybernetikos on Friday March 06, 2020 @07:33PM (#22508773)
It's a great saying, but aren't all large ships these days made out of components that don't float individually?
Re:Cloud Storage for $2 per TB per month by mcdevilkiller on Saturday March 07, 2020 @05:33AM (#22510924)
No, they are mostly made of air, which floats (Partial sarcasm, as that is what makes them float)
Re:Cloud Storage for $2 per TB per month by zchrykng on Friday March 06, 2020 @11:03PM (#22509652)
I mean, the concept actually works, but you have to understand what is actually going into the bundled product. There is no reason you couldn't bundle 100 million in mortgages that had a 10% default risk and sell 10 million of that bundle as AAA.

It is when you just start bundling everything and then saying the entire thing is AAA that the trouble starts.

Re:Cloud Storage for $2 per TB per month by statictype on Friday March 06, 2020 @08:02PM (#22508923)
Except the people who put those sub prime mortgages in probably didn’t apply rigorous statistical analysis to their situation.
Re:Cloud Storage for $2 per TB per month by Aunche on Friday March 06, 2020 @05:24PM (#22507920)
Redundancy is kind of meaningless unless if you have sufficient diversity. If all your replicas are located on the same rack, you're screwed if there is a minor disaster like a sprinkler going off.
Re:Cloud Storage for $2 per TB per month by teknopurge on Saturday March 07, 2020 @12:16AM (#22509894)
This is why the post is a red-herring. You cannot use storage if you cannot get to it. Talking about the price of disk space/mth is absolutely useless without talking about the bandwidth required to use it, which means, if you want to go big-boys-math, 95% bandwidth and peering agreement pricing. That $1.50 with all the network chatter between hosts for replication/repair/etc., not to mention actually _downloading_ of the data from the Sia network will take that $1.50/TB/mth to $70/TB/mth.

Talking about the disk-space pricing is, IMO, disingenuous. Talk about the math for an all-in use-case of the service. I appreciate what the OP is going for, however, hand-waving around uncomfortable critical items like bandwidth cost for the Sia network is not a good look.

Re:Cloud Storage for $2 per TB per month by Dylan16807 on Saturday March 07, 2020 @02:04AM (#22510254)
> not to mention actually _downloading_ of the data from the Sia network will take that $1.50/TB/mth to $70/TB/mth

You really need to back that math up.

Here's my math: 10mbps is over 3TB per month at max utilization, so let's say it's good enough for either 2TB throughput with semi-even use, or 1TB throughput if the use is really focused on part of the day.

When buying transit in bulk in the US, Google tells me that 10mbps was very roughly $8.50 a month in 2017, $4.50 a month in 2018, and $2.50 a month in 2019.

Are you expecting the data to be rebuilt every single day or something?

Re:Cloud Storage for $2 per TB per month by im3w1l on Friday March 06, 2020 @05:53PM (#22508126)
You will be able to boast of 99.9999% uptime up until that sprinkler goes of. And with luck, it could be decades until it does.
Re:Cloud Storage for $2 per TB per month by toolslive on Saturday March 07, 2020 @05:09AM (#22510870)
OTOH, your object storage can be even cheaper if you don't do Reed-Solomon erasure coding, but use rateless erasure codes. For example, online codes[1] have been used by Amplidata to have more reliable storage with lower overhead. There are downsides however (no partial reads, no mutability, ....)

[1] https://en.wikipedia.org/wiki/Online_codes

[dead]
Re:Cloud Storage for $2 per TB per month by justinmeiners on Friday March 06, 2020 @08:49PM (#22509176)
You're taking "cutting corners" out of context. He is describing how individual nodes don't need to run at the standards of regular data servers and are hence cheaper, but in aggregate provide a reliable service.

> So, I assume you’re quite young.

So you do you have a technical criticism?

Re:Cloud Storage for $2 per TB per month by chx on Friday March 06, 2020 @04:23PM (#22507442)
> I didn't read all their math but I expect their final result to be off by a factor of 2-5x.

Can't be more than 2.5 because Backblaze B2 already gives you $5/TB/Mo.

Re:Cloud Storage for $2 per TB per month by TrueDuality on Friday March 06, 2020 @04:48PM (#22507651)
Backblaze is operating at their own economy of scale with dedicated deals with suppliers, custom bare bones hardware, optimized processes, etc. It's always more expensive starting out as a little company for raw hardware and processes until you're big enough and mature enough to get the deals and processes in place.

That is also not the only service that Backblaze offers and wasn't their first. It could be that B2 is simply a way for them to offset their cost for extra capacity and are running it effectively near-cost for them.

Re:Cloud Storage for $2 per TB per month by Aeolun on Friday March 06, 2020 @06:11PM (#22508253)
If all their other customers on the $5/month unlimited backup plan store less than 1TB, but since it’s really aggressive about backing up everything (no, please don’t back up my steam games) on your computer I think they go over that.
Re:Cloud Storage for $2 per TB per month by Dylan16807 on Saturday March 07, 2020 @02:27AM (#22510333)
Here's the histogram from their AMA: https://i.imgur.com/iVEuwUT.jpg

About a third of the users are under 100GB.

Another third is under 500GB.

13% from 500GB to 1TB.

9% from 1TB to 2TB.

8% from 2TB to 5TB.

5% above that.

And they cited raw costs of over $3.50 per TB. But of course that's with real datacenters and non-shoddy equipment.

Re:Cloud Storage for $2 per TB per month by chx on Friday March 06, 2020 @07:07PM (#22508604)
Don't forget the vast amount of notebooks with SSDs that have less than 1TB storage to begin with.
Re:Cloud Storage for $2 per TB per month by dannyw on Saturday March 07, 2020 @04:39AM (#22510806)
Backblaze deduplicates data right? So it doesn’t really cost them anything to back up your steam games.
Re:Cloud Storage for $2 per TB per month by lucb1e on Saturday March 07, 2020 @05:46AM (#22510963)
They can't deduplicate properly encrypted data, that will have to happen client-side. So while they can deduplicate your data (if you have a copy of a picture in multiple places or maybe even a game installed on multiple machines), it won't work across users.
Re:Cloud Storage for $2 per TB per month by Aeolun on Monday March 09, 2020 @06:31AM (#22524290)
In that case I’d also expect them to skip uploading the data, but everything is uploaded.
Re:Cloud Storage for $2 per TB per month by jan6 on Saturday March 07, 2020 @05:34AM (#22510928)
they'd probably charge you the same anyway, even if THEY get it at basically free ;)
Re:Cloud Storage for $2 per TB per month by throwaway3157 on Friday March 06, 2020 @04:46PM (#22507631)
> Can't be more than 2.5 because Backblaze B2 already gives you $5/TB/Mo.

Well it can be, if they have a lot of inefficiencies. Backblaze could have more experienced engineers who overcame these. I assure you, I can accidentally design a very expensive storage system as I’m not that smart ;)

Re:Cloud Storage for $2 per TB per month by pepemon on Friday March 06, 2020 @10:16PM (#22509502)
Who are you?
Re:Cloud Storage for $2 per TB per month by tgsovlerkhgsel on Friday March 06, 2020 @08:11PM (#22508975)
Scaleway C14 [1] gives the $2/TB/month that the title promises. Now, providing Sia storage space may be more expensive because you have to calculate costs for proof-of-storage (and may not be able to pull off the "cold storage" model, which may also only work in conjunction with a higher-priced low-latency offering), plus potentially more replication than a centralized service uses, but it does indicate that it's probably not completely unrealistic.

[1] https://www.scaleway.com/en/c14-cold-storage/

Re:Cloud Storage for $2 per TB per month by ci5er on Saturday March 07, 2020 @01:05AM (#22510077)
Backblaze doesn't give you a random read/write like DropBox, and wanting your disk drive back takes some time...
Re:Cloud Storage for $2 per TB per month by vikiomega9 on Friday March 06, 2020 @04:08PM (#22507309)
> exabyte-scale storage system

Somewhat of a random question, can you point me to some state of the art research?

Re:Cloud Storage for $2 per TB per month by sydney6 on Friday March 06, 2020 @04:26PM (#22507465)
Storage Systems of this scale are thesedays almost exclusively built around object based storage rather than "legacy" block or file backed solutions. I guess today, min.io is would be the way to go. (to "go".. little pun on the end:)
Re:Cloud Storage for $2 per TB per month by statictype on Friday March 06, 2020 @08:04PM (#22508933)
What does object-based mean? How is an object different from a file (which I presumed was a collection of blocks)?
Re:Cloud Storage for $2 per TB per month by tw04 on Friday March 06, 2020 @11:11PM (#22509686)
It means you access your object through a GUID. Think about it like parking your own car vs. a valet. When you park your own car you need to know the address of the garage you parked in, the floor you were on, and the spot you were in. When you valet park, you hand the attendant a ticket and he brings your car back.

With a standard fileshare, you need to walk the filesystem to retrieve your file - this incurs a ton of metadata overhead. It also means when you've got potentially billions of files in a directory, it can be slooooowww. All the metadata requests also make it very chatty - so doing it over a WAN link tends to be extremely painful if it works at all. Newer versions of SMB and NFS have done a lot to batch the metadata requests but they are still protocols meant to happen at extremely low latency inside a datacenter.

Re:Cloud Storage for $2 per TB per month by toolslive on Saturday March 07, 2020 @03:39AM (#22510605)
Some object stores do this, but aws S3 for example does not. You can list the contents of buckets, nicely sorted by name. You can mimic directory structures if you want.

However, you touched a key point: object stores are all about throughput, not latency. You can store at a GB/s (if you have the pipes), but even checking if an object exists will cost you a few milliseconds.

Re:Cloud Storage for $2 per TB per month by statictype on Saturday March 07, 2020 @02:49AM (#22510427)
Got it. Thanks for explaining.
Re:Cloud Storage for $2 per TB per month by tgsovlerkhgsel on Friday March 06, 2020 @08:13PM (#22508983)
My guess: No random write access to objects, you can (at best) append-only but often you can only append until the object is finalized, and cannot read it until it is finalized.
Re:Cloud Storage for $2 per TB per month by toolslive on Saturday March 07, 2020 @03:32AM (#22510573)
it's like a key-value store, (or a dictionary). However, the values are objects (big blobs of data). This means you can't update parts of objects without rewriting the blob. However, most of the object stores offer metadata operations (move, tag, ...), concat of n objects into 1 and partial reads.
Re:Cloud Storage for $2 per TB per month by londons_explore on Friday March 06, 2020 @06:24PM (#22508342)
Google's clusters are all block and file based (depending on what layer you want to use them at)
Re:Cloud Storage for $2 per TB per month by ghostpepper on Friday March 06, 2020 @04:17PM (#22507387)
I am not the op nor an expert on this, but I think https://ceph.io/ targets clusters of this scale.
Re:Cloud Storage for $2 per TB per month by Mave83 on Saturday March 07, 2020 @03:41AM (#22510623)
Yes, with croit.io you can manage Ceph clusters with ease, reducing Laber costs and increase reliability.

We build clusters of low PB scale that have a TCO with everything from labor to hardware, from financing to electric, from routers to cables and can be run below 3€/TB. For that you can store data in Block(rbd, iscsi), Objekt(S3, swift) or Filestorage(CephFS, NFS, SMB), high available on Supermicro hardware in any datacenter worldwide.

Feel free to contact us or use our free community edition to start your own cluster.

Re:Cloud Storage for $2 per TB per month by z3t4 on Friday March 06, 2020 @04:27PM (#22507479)
Storage can be far cheaper when decentralized. Sending data over the Atlantic is super expensive compared to LAN networking. Almost all content providers peer with ISP's with onsite hardware. But why stop there, put the "racks" in ppl's basements. Data storage is very compact now a days, you can probably fit 100 TB is a shoe-box.
Re:Cloud Storage for $2 per TB per month by jan6 on Saturday March 07, 2020 @05:41AM (#22510950)
that'd need trust that they won't run off with your drives, their basement won't get flooded, there aren't power outages often, they have good protections against surges and whatnot... which at ANY datacenter, is automatically included ;)
Re:Cloud Storage for $2 per TB per month by Dylan16807 on Saturday March 07, 2020 @08:56PM (#22515769)
You need to trust that not too many of them will have that happen at the same time.

Which is not that hard to do.

And all those protections in a datacenter are far from free.

Re:Cloud Storage for $2 per TB per month by winfred on Friday March 06, 2020 @06:23PM (#22508335)
>I didn't read all their math but I expect their final result to be off by a factor of 2-5x.

I looked at their parts list and it's obvious they aren't serious. CPU is missing, memory is missing, SAS to SATA cables, but no SAS controller, no mounting for the system board. Low effort at best.

Re:Cloud Storage for $2 per TB per month by prostheticvamp on Friday March 06, 2020 @10:48PM (#22509615)
It seems you’re getting heavily downvoted. I think it’s because CPU and memory are not missing.
Re:Cloud Storage for $2 per TB per month by winfred on Tuesday March 10, 2020 @03:35PM (#22538761)
That wasn't on that page when I opened it, you think I'd miss that? They were reading the comments here and updated the page after I wrote the comment.

The price was $4700 something, now it's $4945.

They simply smooth over it with this:

"So we will be using a rig cost of $4500 in our spreadsheet."

That way their overall math doesn't change. Should get banned for doing things like that. These guys are up to no good.

Re:Cloud Storage for $2 per TB per month by notyourday on Friday March 06, 2020 @04:20PM (#22507418)
We have done this calculation and even if you put your gear into Equinix/Digital Realty in the most expensive places and use Backblaze-type setup ( which is not optimized and buying retail) bringing 10Gbit to every 4U the price for double-writes at 5TB disks are $10/year per TB.
Re:Cloud Storage for $2 per TB per month by FalconSensei on Friday March 06, 2020 @07:25PM (#22508732)
> Hard drives are a surprisingly low percentage of the cost of a storage system

THIS! There's no use of a 2TB storage if you can't upload/download this amount each month

Re:Cloud Storage for $2 per TB per month by tlb on Friday March 06, 2020 @07:54PM (#22508888)
Many businesses have TB of data they upload and keep for several years, looking at it once or twice over that time if ever. For these you could provision monthly bandwidth as low as 1/30th of the total storage.
Re:Cloud Storage for $2 per TB per month by Dylan16807 on Saturday March 07, 2020 @09:04PM (#22515806)
It says right there in the article that bandwidth is charged for separately. So you just buy a line of appropriate size, based on actual usage measurements. And each one of these rigs only needs a gigabit connection to upload/download its full capacity each month, which means the network equipment costs can be minimal.
Re:Cloud Storage for $2 per TB per month by jan6 on Saturday March 07, 2020 @05:44AM (#22510956)
have you ever heard of backups and archiving? ;) there are lots of scenarios when you'd want to write some amount, but not read it back for a while, if ever...
Re:Cloud Storage for $2 per TB per month by late2part on Friday March 06, 2020 @06:53PM (#22508509)
I agree with you. Not 2-5x but they are rounding down on costs and optimistic on risks.