Counter-point: You know that you're the dominant player. See: .psd, .pdf, .xslx. Not particularly good file types, yet widely supported by competitor products.
Every file format accumulates cruft over thirty years, especially when you have hundreds of millions of users and you have to expand the product for use cases the original developers never imagined. But that doesn’t mean the success wasn’t justified.
A very popular file format pretty much defines the semantics and feature set for that category in everyone's mind, and if you build around those features, then you can probably expect good compatibility.
Nobody thinks about the actual on disk data layout, they think about standardization and semantics.
I rather like PDF, although it doesn't seem to be well suited for 500MB scans of old books and the like, they really seem to bog down on older mobile devices.
It’s by no means impossible to do that yourself, but it costs a lot more in time and upfront expense.
Of course, I lost that script when the server crashed, the one thing I didn't back up properly.
What does that mean? If I have 1 million objects, I loose 100 per year?
If you had light bulbs that lasted 1,000 hrs on average, and you had 10k light bulbs, and turned them all on at once, then they would all last 1,000 hours on average. Some would die earlier and some later, but the top line number does not tell you anything about the distribution, only the average (mean). That’s what MTTF is; the mean time for a given part to where it has a greater likelihood to have failed by then vs not. It doesn’t tell you if the distribution of light bulbs burning out is 10 hrs or 500 hrs wide. it’s the latter, you’ll start seeing bulbs out within 750 hrs, but if the former it’d be 995 hrs before anything burned out.
If you have ten million objects, you should lose one every 10k years or so.
S3's original implementation might have only had 11 9s, and it just doesn't make sense to keep updating this number, beyond a certain point it's just meaningless
Like "we have 20 nines" "oh yeah, well we have 30 nines!"
To give an example of why this is the case, if you go from a 10:20 sharding scheme to a 20:40 sharding scheme, your storage overhead is roughly the same (2x), but you have doubled the number of nines
So it's quite easy to get a ton of theoretical 9s with erasure coding
You could unknowingly upload a corrupted file, for example. By the time you discover that, there may not be a clear record of operations on that object. (Yes, you can record S3 data plane events but that’s not the point.)
Only the customer would know if their data is intact, and only the customer can ensure that.
The best S3 (or any storage system) can do is say “this is exactly what was uploaded”.
And you can overwrite files in S3 with the appropriate privileges. S3 will do what you ask if you have the proper credentials.
Otherwise, S3 is designed to be self-healing with erasure encoding and storing copies in at least two data centers per region.
AWS probably doesn’t want those risks and ambiguities.
Over the years working on countless projects I’ve come to realize that the more “easy” something looks to an end user, the more work it took to make it that way. It takes a lot of work to create and polish something to the state where you’d call it graceful, elegant and beautiful.
There are exceptions for sure, but often times hidden under every delightful interface is an iceberg of complexity. When something “just works” you know there was a hell of a lot of effort that went into making it so.
What do you see as special for S3? Isn’t it just another bucket?
Moral of the story: the "technical part" of things is not the end of the story
Alternative moral: The 3-2-1 backup rule of thumb is still alive and well (is your cloud account a single point of failure?)
Related: https://www.infoworld.com/article/2179073/murder-in-the-amaz...
One of the things that shocks me about the system is the level of object durability. A few years ago I was taking an AWS certification course and learned that their durability number means that one can expect to loose data about once every 10,000 years. Since then anytime I talk about S3's durability I bring up that example and it always seems to convey the point for a layperson.
And it's "simplicity" is truly elegant. When I first started using S3 I thought of it as a dumb storage location but as I learned more I realized that it had some wild features that they all managed to hide properly so when you start it looks like a simple system but you can gradually get deeper and deeper until your S3 bucket is doing some pretty sophisticated stuff.
Last thing I'll say is, you know your API is good when "S3 compatable API" is a selling point of your competitors.