That's like saying "how to get to the moon is obvious: traveling"

Thank you for setting me up for this...

It's not exactly rocket science.

Haha, good one!

I still feel like you're underselling the article however.

Is obviously ultimately parallelism, but parallelism is hard at scale - because things often don't scale - and incorrect parallelism can even make things slower. And it's not always obvious why something gets slower by parallelism.

As a dumb example, if you have a fictional HDD with one disk and one head, you have two straightforward options to optimize performance:

Make sure only one file is read at the same time (otherwise the disk will keep seeking back and forth)

Make sure the file is persisted in a way that you're only accessing one sector, never entering the situation in which it would seek back and forth.

Ofc, that can be dumped down to "parallelism", because this is inherently a question about how to parallelize... But it's also ignoring that that's what is being elaborated on: ways s3 used to enable parallelism

I dunno, the article's tl;dr is just parallelism.

Data gets split into redundant copies, and is rebalanced in response to hot spots.

Everything in this article is the obvious answer you'd expect.

It's not really "redundant copies". It's erasure coding (ie, your data is the solution of an overdetermined system of equations).

That’s just fractional redundant copies.

And "fractional redundant copies" is way less obvious.

The fractional part isn't helping them serve data any faster. To the contrary, it actually reduces the speed from parallelism. E.g. a 5:9 scheme only achieves 1.8x throughput, whereas straight-up triple redundancy would achieve 3x.

It just saves AWS money is all, by achieving greater redundancy with less disk usage.

You're right if you're only looking at peak sequential throughput. However, and this is the part that the author could have emphasized more, the impressive part is their strategy for dealing with disk access latency to improve random read throughput.

They shard the data as you might expect of a RAID, 5, 6, etc array and the distributed parity solves the problem of failure tolerance as you would expect and also improves bandwidth via parallelism as you describe.

The interesting part is their best strategy for sharding the data: plain-old-simple random. The decision of which disks and at which sectors to shard the data is done at random, and this creates the best change that at least one of the two copies of data can be accessed with much lower latency (~1ms instead of ~8ms).

The most crude, simple approach turns out to give them the best mileage. There's something vaguely poetic about it, an aesthetic beauty reminiscent of Euler's Identity or the solution to the Basel Problem; a very simple statement with powerful implications.

I generally don't think about storage I/O speed at that scale (I mean really who does?). I once used a RAID0 to store data to HDDs faster, but that was a long time ago.

I would have naively guessed an interesting caching system, and to some degree tiers of storage for hot vs cold objects.

It was obvious after I read the article that parallelism was a great choice, but I definitely hadn't considered the detailed scheme of S3, or the error correction it used. Parallelism is the one word summary, but the details made the article worth reading. I bet minio also has a similar scaling story: parallelism.

My homelab servers all have raidz out of 3 nvme drives for this reason: higher parallelism without loosing redundancy.

> I would have naively guessed an interesting caching system, and to some degree tiers of storage for hot vs cold objects.

Caching in this scenario usually done outside of S3 in something like Cloudfront

If you’re curious about this at home, try Ceph in Proxmox.

Unless you have a large cluster with many tens of nodes/OSDs (and who does in a homelab?) then using Ceph is a bad idea (I've run large Ceph clusters at previous jobs).

Why is it a bad idea?

Traditional storage has one RAID Controller with a lot of disks hanging off it, traditional storage servers have one server fronting to the network with a lot of disks inside it, traditional SANs have one head with a couple of controllers and a lot of disk shelves hanging off the back, this design bottlenecks at the single front end.

Ceph is designed not to have a single front end, any node in the cluster can take data in and relay it to the target node where the data is stored. To avoid having a single point of failure where all the nodes look to one lookup table to say where the data lives, they all use the algorithm to choose where the data lives. This is guided by admin rules known as the CRUSH Map where to say things like:

- these storage servers are grouped together by some label (e.g. into a rack, using the same power feed, in a data centre, same site)

- I want N copies of data, and separated over different redundancy / failure boundaries like different racks or different sites.

The result of this design is a massively parallel system that can ingest a lot of data in parallel streaming using every node’s network bandwidth, but quite a lot of internal bandwidth shuffling data around, and quite slow in latency because there’s a layer of calculation and mapping and inter-node network traffic for every block read/written.

The result of this design is that the resting place for a piece of data is deterministic, like a hash of the data guided by placement rules, so that every node follows the algorithm and arrives at the same resulting location where the data needs to go/come from and there is no single controller which can remap the storage “ohh this location is unbalanced and filling up, put it there instead“. As a result of that, the cluster can’t really run with different size disks very well or very easily, and it’s “as full as the fullest disk” - one full disk will put the entire cluster into read-only mode until you fix it. So it runs best with raw storage around half full or less, and keep space free to move data in case of failure - if you said store data per rack, a whole rack’s worth of rebuilding might be needed.

It’s got a design where the units of storage are OSDs which correspond roughly to disks/partitions, each one has a daemon controlling it, on top of that Ceph pulled the storage together as RADOS - reliable autonomic distributed object store - and on top of that you can layer user-visible storage such as CephFS filesystem, Amazon S3 compatible object storage, or a layer that presents as a block device which can be formatted with XFS/etc. so there’s another layer there which doesn’t help latency.

And it needs monitor daemons and management daemons overseeing the whole cluster to keep track of failed storage units and make that available to all nodes, and management/monitoring services.

So overall it has a high administrative complexity to setup, and needs a lot of hardware/storage to get the best out of it. Any small use case will be better off with a traditional SAN or Linux JBOD (just a box of disks) server, easier to manage and higher performance, lower latency, and then move to two file servers for network parallelism or sync between them.

Ceph can run with HDDs but has a separate place for metadata to be on SSD but if that’s not sized well then it spills over to HDD and performance worsens. I inherited some Ceph management work and it’s gone into my list of things I respect from a distance but don’t enjoy. Sad because it’s a very interesting system, but there was no management appetite for investing in it so it was overloaded, underperforming, and caused many problems.

It can be all set up on three nodes I think, so you can play with it, but I wouldn’t think that’sa good production design. It needs a lot of nodes and some high effort setup to really spread itself out and you need a use case that benefits from petabytes of storage and streaming writes in parallel - there are some talks on YouTube by people who managed and upgraded it at CERN as targets of particle accelerators data which are quite interesting.

While most of what you speak of re Ceph is correct, I want to strongly disagree with your view of not filling up Ceph above 66%. It really depends on implementation details. If you have 10 nodes, yeah then maybe that's a good rule of thumb. But if you're running 100 or 1000 nodes, there's no reason to waste so much raw capacity.

With upmap and balancer it is very easy to run a Ceph cluster where every single node/disk is within 1-1.5% of the average raw utilization of the cluster. Yes, you need room for failures, but on a large cluster it doesn't require much.

80% is definitely achievable, 85% should be as well on larger clusters.

Also re scale, depending on how small we're talking of course, but I'd rather have a small Ceph cluster with 5-10 tiny nodes than a single Linux server with LVM if I care about uptime. It makes scheduled maintenances much easier, also a disk failure on a regular server means RAID group (or ZFS/btrfs?) rebuild. With Ceph, even at fairly modest scale you can have very fast recovery times.

Source, I've been running production workloads on Ceph at fortune-50 companies for more than a decade, and yes I'm biased towards Ceph.

I defer to your experience and agree that it really depends on implementation details (and design). I've only worked on a couple of Ceph clusters built by someone else who left, around 1-2PB, 100-150 OSDs, <25 hosts, and not all the same disks in them. They started falling over because some OSDs filled up, and I had to quickly learn about upmap and rebalancing. I don't remember how full they were, but numbers around 75-85% were involved so I'm getting nervous around 75% from my experiences. We suddenly commit 20TB of backup data and that's a 2% swing. It was a regular pain in the neck, stress point, and creaking, amateurishly managed, under-invested Ceph cluster problems caused several outages and some data corruption. Just having some more free space slack in it would have spared us.[1]

That whole situation is probably easier the bigger the cluster gets; any system with three "units" that has to tolerate one failing can only have 66% usable. With a hundred "units" then 99% are usable. Too much free space is only wasting money, too full is a service down disaster, for that reason I would prefer to err towards the side of too much free rather than too little.

Other than Ceph I've only worked on systems where one disk failure needs one hotspare disk to rebuild, anything else is handled by a separate backup and DR plan. With Ceph, depending on the design it might need free space to handle a host or rack failure, and that's pretty new to me and also leads me to prefer more free space rather than less. With a hundred "units" of storage grouped into 5 failure domains then only 80% is usable, again probably better with scale and experienced design.

If I had 10,000 nodes I'd rather 10,100 nodes and better sleep than playing "how close to full can I get this thing" and constantly on edge waiting for a problem which takes down a 10,000 node cluster and all the things that needed such a big cluster. I'm probably taking some advice from Reddit threads talking about 3-node Ceph/Proxmox setups which say 66% and YouTube videos talking about Ceph at CERN - in those I think their use case is a bursty massive dump of particle accelerator data to ingest, followed by a quieter period of read-heavy analysis and reporting, so they need to keep enough free space for large swings. My company's use case was more backup data churn, lower peaks, less tidal, quite predictable, and we did run much fuller than 66%. We're now down below 50% used as we migrate away, and they're much more stable.

[1] it didn't help that we had nobody familiar with Ceph once the builder had left, and these had been running a long time and partially upgraded through different versions, and had one-of-everything; some S3 storage, some CephFS, some RBDs with XFS to use block cloning, some N+1 pools, some Erasure Coding pools, some physical hardware and some virtual machines, some Docker containerised services but not all, multiple frontends hooked together by password based SSH, and no management will to invest or pay for support/consultants, some parts running over IPv6 and some over IPv4, none with DNS names, some front-ends with redundant multiple back end links, others with only one. A well-designed, well-planned, management-supported cluster with skilled admins can likely run with finer tolerances.

Disagree. I run Ceph in Proxmox and have for years on a small cluster of 3 used R620 servers without any SSDs.

It’s just worked. I’ve lost two of the machines due to memory failures at two different points in time and the k8s clusters sitting on top didn’t fail, even the Postgres databases running with cnpg remained ready and available during both hardware failures.

Oh sure it works, not denying that. My point is that performance isn't great and if you only have a small cluster then it doesn't take much to make everything fall over because your failure domains are huge (in your case, you only have 3).

But then to offset the above, it also depends on how important your environment is; homelabs don't usually require five nines.

I am a big Proxmox fan but I dislike how easy it makes Ceph to run (or rather, how it appears to be easy). Ceph can fail in so many ways (I've seen a lot of them) and most people who set a Ceph cluster up through the UI are going to have a hard time recovering their data when things go south.

AWS themselves have bragged that the biggest S3 buckets are striped across over 1 million hard drives. This doesn't mean they are using all of the space of all these drives, because one of the key concepts of S3 is to average IO of many customers over many drives.

RAID doesn’t exactly make writes faster, it can actually be slower. Depends on if you are using RAID for mirroring or sharding. When you mirror, writes are slower since you have to write to all disks.

He explicitly mentioned RAID0 though :)

I think the article's title question is a bit misleading because it focuses on peak throughput for S3 as a whole. The interesting question is "How can the throughput for a GET exceed the throughput of an HDD?"

If you just replicated, you could still get big throughput for S3 as a whole by doing many reads that target different HDDs. But you'd still be limited to max HDD throughput * number of GETs. S3 is not so limited, and that's interesting and non-obvious!

Millions of hard drives cumulatively has enormous IO bandwidth.

I can only imagine the counts of those hdds