Cooking isn't hard either but I still go to restaurants, and don't go around laughing at patrons telling them that they never bothered to learn how to cook if they are there.
In any case one does it professional IT done right is costly but cheaper than e.g. IT without offsite backup in case of a disaster.
Next time you see someone paying for something you think it's easy to learn, that the "zeitgeist told them not to learn", go up to them and say what I quoted and you'll see real fast if it's condescending or not.
The difference in situations here is literally that on the one hand large corporations have spent money to market the idea that their services are essential because the alternative is incredibly difficult. On the other hand, McDonalds, KFC, Burger King, "Big Restaurant" etc have not spent large quantities of money trying to persuade the populace that cooking at home is incredibly difficult. It's possible to sell someone a convenience without also telling that person that it's the only real option and this is what the person you're replying to is highlighting.
It's not condescending, it's the reality of the situation. Yes in the examples you've described it would be condescending but in the situation we're actually talking about there are actually large numbers of people who genuinely believe (as a result of successful marketing campaigns) that the equivalent to cooking your own meals at home is incredibly dangerous, difficult, expensive and error prone.
They haven't because it's a ridiculous proposition. The same way cloud providers would be laughed out of the room if they tried to say the same thing and they didn't actually have a better product with trade-offs that are genuinely better for some use cases. The fact that you believe that everyone using cloud products is either too lazy to learn or has been fooled by marketing because they can't think for themselves, unlike you who see the light, is the condescending part. Mind you, on the face of overwhelming data showing most companies have, or are moving large parts of their infrastructure to the cloud, so everyone else are dummies either too lazy or too stupid to see what you can see.
I never said anyone was too lazy, just misinformed about the benefits, and it's a more far fetched statement to claim that marketing doesn't work than it is to state that it does work.
As evidenced by some people in this thread claiming that not using S3 would be basically impossible (despite a large number of popular distributions literally doing exactly that) there are clearly plenty of people who have fallen for the marketing.
Moreover, you're the one implying that falling for marketing is something only people who can't think for themselves do, that's also not what I'm saying. I think everyone is susceptible to marketing, and in this case, clearly a lot of intelligent people who are otherwise capable of thinking for themselves have fallen for this blindspot.
I think you are the one being condescending here.
“Assume good faith”[1]
Not sure about that. There appear to be just as many people in IT, it's just that they're not that technical anymore, mostly producing policy documents and pushing tickets around. The actual work is outsourced to the cloud, SaaS products and outside consultants (for duct taping the mishmash of SaaS products together). But that may be just my lacking understanding of my strange little part of the globe.
Things don't fail as much as they used to, at least not from using something mature as minio and of course hardware can fail as they do specifically spinning rust, which is non trivial to replace if you have hot-swap capable servers in the first place.
I don't possess any decade worth expertise in either minio or running this kind of setup so personally I believe you are exaggerating that this is even worth calling challenge at all.
But that's just my take :)
It’s so easy, after all.
Pretending to know how much work something is when it's not your area of expertise is a bit silly.
Actually, yes, it is easy, after all :)
It's like comparing leasing a Mercedes-Benz with full service and all inclusive insurance to building your own bicycle. Some people don't need the car, sure, but it's not the same thing.
Although having to spend some time to attain good enough availability/durability is not a big deal when the rush to launch was over long ago. Unlike in your example, you can turn the bicycle into a car.
All it needs to do is to get you the data faster than the compilation itself - most of the time - to be useful.
I also wonder at the original thinking, given the use case. Cache usually has high ratio of download to upload and doesn't need high reliability. S3 doesn't excel at the costs of fetching the data from it, at all.
Do you ACTUALLY think you can communicate with people at Amazon who truly know what's going on? If so, why do we see countless stories about people who have problems with Amazon, Google, et cetera, who can't communicate with them? Do you know of some special, magic way to get the skilled people to talk with you that the rest of us don't?
You act like being able to call Amazon, even if you could get someone on the line who knows something, is somehow better than having even just one good systems administrator. All of your ridiculous assumptions go right out the window when you have a sysadmin build, test and deploy your own hardware that you (via your sysadmin) have complete control over.
Security? You can't secure Amazon. You literally can't. If you wanted to, you'd have to buy the company, then you'd have to evaluate your whole S3 staff. If you hire a systems administrator, you have to vet your systems administrator and anyone else who has admin access, not a collection of tens of thousands of employees across the planet, any of whom could be stupid enough to visit the wrong web site using the wrong browser on the same computer as their keys.
Availability? My side projects have better uptime than Amazon.
Durability? Two geographically disparate locations and LTO, plus many mirrors, make things plenty durable.
When you're not a systems administrator, you can be forgiven for not understanding the big picture, but not for acting like you do.
And for the point in question, you don't need to talk to anyone, the point is there's thousands of datacenter, networking, app-level layers of staff all working to make service stable for you without you even knowing. I don't need to talk to anyone if an S3 server, rack, or even full DC goes down.
If this is threatening in a serious countenance, our growth and ability to conduct our project. I'm not sure if we need all the 9s of durability of S3, given that people already lost all their buckets on S3.
We do not have the money to store 5x or 6x the contents of the cache.
So we much prefer to have control, and we have enough skills to run all that shit easily, the problem is that the skilled people are in rare availability and are usually working on harder problems than running that. So ultimately, this is a balance problem.
Disclaimer: NixOS developer, 23.05 Release Manager.
Half PB is a bit more than that, and I guess it is expected to grow... in 2023 it is not realistic to throw such storage around for free (+mirrors). It will be realistic in a few years considering that the storage prices have been in free fall for the past 30 years, but that time hasn't came yet.
(As a maybe interesting comparison, Debian's mirror is around half TB: https://people.debian.org/~koster/www/mirror/size )
https://www.hetzner.com/dedicated-rootserver/matrix-sx
(You would need 4 SX294s for half a petabyte with 2x replication)
And put ceph on them which gives you an S3-compatible API:
https://docs.ceph.com/en/quincy/radosgw/index.html
It's not trivial to set up but once you do it's much cheaper than AWS. It's even cheaper if you can find the machines in the server auction. There are some 150 TB machines on there right now for 174 euro/month.
Also, your math does not quite add up if you want to store 425TiB. You'd be filling up the Ceph cluster to 95% which would be impossible (you'll start losing OSDs way before that).
If this was me, I would definitely do this on Ceph, but I would opt for a more traditional erasure coding setup of 8+3. For that, you'd need 12x SX134s as a minimum starting point.
I suppose you could start with an EC profile of 2+2 or 4+2...
I do have a serious question on Hetzner:
Do they offer any private networking options? That page only lists a single 1 Gbit connection. I don't think that would work well with Ceph. It's doable, sure, but I would want to support such a cluster, not with 160TB raw per box.
Doesn't seem like it for their dedicated servers, not 100% sure though.
Their docs indicate that upgrading the 1Gbit ethernet link to 10GBe is an available option:
https://docs.hetzner.com/robot/dedicated-server/network/10g-...
Extending with additional NICs should be possible from what I've read but not used myself.
Here's what you do:
1. Buy two servers equipped with a large amount of storage. There are many providers[1][2] that offer this. (You could technically do it with one, but two allows you to have high availability and prevent data loss in case of accidents, imagine `rm -rf *`.)
2. Use Minio to set up erasure-coded storage on one of the servers, and replicate objects[3] to the other server.
4. Use Cloudflare (because of their open source sponsorship program[4]) to serve the objects from the primary server, and when performing maintenance tasks, fall back to the secondary.
You could also serve objects directly out of the primary/secondary, but using Cloudflare takes care of low-effort DDoS attacks by hiding the origin, and provides some caching which helps prevent hitting the egress limits of the server provider.
[1] https://www.netcup.eu/vserver/vstorage.php
[2] https://www.hetzner.com/dedicated-rootserver
[3] https://blog.min.io/active-active-replication/
[4] https://blog.cloudflare.com/cloudflare-new-oss-sponsorships-...