But secondly, offline doesn't usually mean dead and gone, with unstable datacenters like this they are usually back online before the user has lost a full 25% of their redundancy.
Row level and rack level outages are handled by data randomization. The entire Sia system heavily depends on probabilistic techniques, both on the renting and hosting side. Row level failures will take out some of your data, but nobody should be disproportionately impacted by a cluster failure.
On Sia, each piece is at a different site. So 64-of-96 implies that each chunk of data (96 pieces to a chunk) is located in 96 different places. This doesn't help with the geo-bandwidth, but as discussed above there are other techniques to handle that.
Surprisingly, bandwidth pricing on the Sia network is even cheaper than storage pricing relative to centralized competition. That's a lot harder to model at scale though, so we aren't as confident the Sia bandwidth pricing will hold up at $1 / TB in the long term.
And technically, most of this stuff is customizable per-customer. If your particular use case has a different optimal parameterization, it's fairly easy to tune your client to suit your particular needs.
I mean this is basically the moment where I would expect every systems designer on HN coming out of the woods and crushing Sia into the ground, if there were, in fact, any ground at all to crush Sia into.
Is this actually legit? If so, where is the rejoicing? What am I missing?
At this point the cautious skeptic will be thinking "hmm, maybe there's something to this", not necessarily full on rejoicing.
That said, I agree it does seem promising. If you ever find yourself in need of cheap cloud storage it wouldn't hurt to look into Sia as a possible option.
Engineering is about tradeoffs. I could build a network as good as Google's with infinite money, infinite time, and infinite help. I could design a product as beautify as Apple's with the same lack of limitations. Unfortunately for me, I have limited money, limited time, and limited help. Every systems designer understands that, innately, so isn't rushing out of the woodword because Sia and Dropbox have merely chosen different tradeoffs. That one has IPO'd is uninteresting in the abstract. It's just money after all.
Reed-Solomon encoding adds 50%, of you want 3 block per 2 data blocks. Replicated encoding (not relevant here since this is allow throughput usecase, but necessary if you want to sustain high read throughput) is adding at least 200% (if you want a 3x replication, which I think should be the minimum).
Amazon's goal is to get people to pay a premium to get access to their entire ecosystem of services. They don't optimize the price of each individual service. There's a lot of take it or leave it for their specific offerings. Look at how they used to have reduced redundancy as a cost saver, but don't anymore. If you want corner-cutting storage from Amazon, you get funneled into Glacier now.
So let's look at how glacier deep archive is $.99 per TB per month, with a waiting time. And Google's weird offering where it costs $1.23 per TB per month, with instant access, but you pay a bunch of money to access. That means they can't store it on tapes, but they expect to profit anyway. And they're probably not running that as a loss leader that depends on the archival data being accessed.
Actually, looking closer, is one-zone even storing fewer copies? It's offering the same durability, except in the case of "availability zone destruction". So if they're selling three copies for $10 a TB, then the equivalent price for 64-of-96 would be $5 a TB. Cutting that down with cheaper worse hardware would go a long way to get you toward the $2 goal.
If instead they decide it's the cash cow, then it wouldn't be a stretch to predict that they price it a bit above the competition. The customer has the impression that they can amortize the cost across the other value-adds in the ecosystem. Whether the customer is right or not barely matters to Amazon from a purely financial aspect. All that matters is that they believe it to be true.
There's also the short-term versus the long term strategy. Short term they could do either and I wouldn't be surprised. Long term, I think I would expect the latter.
Glacier Deep Archive is .99/TB/month in US East and EU Ireland and several others, plus $2.50/TB bulk retrieval.
https://cloud.google.com/storage/pricing
Archive Storage is $1.20/TB/month in us-east1 and europe-west1 and several others. But (instant) retrieval is $50/TB, with no options...
A good is example are the Apple iPhone screens. If Apple wants a new competing supplier, they have to invest in new competitors.
This is bad for customers in the long term. Like how Amazon is slowly destroying brick and mortar stores, or having Amazon Basics at the front undercutting other sellers. Monopolies mean you don’t have an open fair market anymore.
I'm explaining a common business move called pay to play, in industries that require very high investments for new competitors. If customers want B to compete, they have to pay B first to ensure B does not make a loss, so everyone wins a little in the end.
So in the case of OP. He needs to finds paying customers first, who are willing to pre-order his service.
Definitely not worth it.
To get out of Amazon entirely, they definitely want to gouge you, but it's not the end of the world. If $24 a year is an acceptable storage price, then Snowball export costing somewhere around $36/TB in bulk isn't too awful. (And if you don't have enough data to fill up a snowball, you can probably smuggle it out through lightsail.)
so many options
From operational experience reports of S3 I've seen out there and discussions [1], including on HN [2], once you reach a large enough object count, S3 is "sorta-reliable". Objects will disappear. The durability claim is not SLA-enforced [3], which is uptime-oriented. The only remedy is a service credit.
[1] https://www.theregister.co.uk/2018/07/19/data_durability_sta...
I didn’t think much about it, but I guess I was really unlucky.
Because of AWS scale it cannot be the cheapest. Amazon cannot use odd lots or small lots, which means that it has very few possible suppliers and those suppliers will never be the cheapest.
Same goes for the network infrastructure. While it is possible buy 100Gbit/sec at a throw away price to because some sales person needs to make his numbers and neither he nor his director of sales thinks Kmod Hosting would be able to fill the pipe it is buying, Amazon would need 900Gbit/sec for their exit and no one at the vendor is going to blink at making Amazon pay higher price than Kmod Hosting.
If you are able to take advantage of sales and clearance deals here and there, you are very likely to get a lower price than even the biggest buyers.
E.g bandwidth costs on AWS are high enough that if you actually serve up lots of data from S3 you can typically afford to rent servers to cache all of it 'in front' of AWS and still save a ton of money.
S3 only gets close to competitive if you never access the data from outside of AWS.
Which gets to the point: If you use an AWS service like S3, you pretty much has to use other AWS services if you want the cost to be even somewhat reasonable in aggregate.
They don't need to compete on cost, because once they get you to buy into one set of services, moving any one set of services off AWS gets more painful and/or costly, and a full migration looks too scary for most people.
S3 will never be priced to undercut anything but big players for that reason. They need to be competitive with Google and Azure, because those guys can offer to offset transitioning costs and generally aggressively target AWS customers.
A small player isn't the same threat even if substantially cheaper.
This post is intended to address people who do not feel that decentralization can be cost effective at scale.
https://support.sia.tech/article/dk91b0eibc-welcome-to-sia - this support article is also a good introduction to the network and why it was built
Also take a look at Skynet, a file sharing protocol built on top of Sia: https://siasky.net/
Reliability, redundancy, disaster recovery, cost cutting, and absolute focus on execution!
As someone who has worked with a couple of market leaders in their respective fields i want to disssaude anyone who would listen of this notion. NO, the market leaders aren't doing things optimally. Some things are downright stupid.
That puts the cost of 192 TB at more like $6240, not $4945. Could be less if you find a good deal on mini-SAS PCIE cards, but still going to be substantially higher than $4945.
In theory something like an Epyc 3000-series with lots of PCIe or onboard SATA that supports port multipliers would work great, but I don't think anyone actually makes that.
To me a big part of computer science is abstracting away unreliable details to make them seem reliable.
Now of course that assumes they are uncorrelated, but since Sia nodes are distributed across the internet, that's likely as opposed to multiple servers at a few data centers like AWS.
Turns out mortgage bonds tend to be correlated.
Tying two bricks together doesn't make them float.
It is when you just start bundling everything and then saying the entire thing is AAA that the trouble starts.
Talking about the disk-space pricing is, IMO, disingenuous. Talk about the math for an all-in use-case of the service. I appreciate what the OP is going for, however, hand-waving around uncomfortable critical items like bandwidth cost for the Sia network is not a good look.
You really need to back that math up.
Here's my math: 10mbps is over 3TB per month at max utilization, so let's say it's good enough for either 2TB throughput with semi-even use, or 1TB throughput if the use is really focused on part of the day.
When buying transit in bulk in the US, Google tells me that 10mbps was very roughly $8.50 a month in 2017, $4.50 a month in 2018, and $2.50 a month in 2019.
Are you expecting the data to be rebuilt every single day or something?
> So, I assume you’re quite young.
So you do you have a technical criticism?
Sia systems don't need a ton of networking. I ran the networking buildout costs by some networking people, and again it comes down to cutting corners. If you only need 10 gbps per rack, if you don't mind having extra milliseconds added, etc, you can get away with very scrappy setups. The whole point is that it's not a highly reliable facility.