S3 API compatibility is there because creators of open source "data engineering" tools have in the early days rushed to develop their tools around the S3 API. I can make the case that the need for a properly abstracted storage layer should have been obvious, but in that early gold rush no one would have listened to such reason. Nowadays vendors are stuck shipping S3 compatibility layers in order to avoid locking out tools that cannot work without S3.

If you think I'm wrong, imagine for a moment what life would be like if every RDBMS client in existence got written in terms of Oracle's wire protocol, back when Oracle's RDBMS offering dominated its respective market. Thankfully that parallel universe is somebody else's problem right now, but it could have happened.

Not at all. It is there because AWS's S3 became the defacto solution for storage and because they were first AND also because of popularity and how well it works.

Your Oracle example is flawed for many reasons. SQL standard pre-dates Oracle and I am not even sure what you mean by every RDBMS client got written in Oracle's wire protocol. Ingres also predates Oracle. Are you implying that S3 API is there because they were the first? Nothing else?

https://en.wikipedia.org/wiki/Ingres_(database)#Early_histor...

SQL standard:

Developer ISO/IEC First appeared 1974; 45 years ago

Oracle first release: Oracle v2 2.3 1979

And also, it is allowed and trivial to implement S3 APIs because it is well documented and available for anybody. You are also implying that S3 API is not a "properly abstracted storage layer"?

Save the Wikipedia links: surely you remember the time when database selection boiled down to "Oracle or bust". I'm not implying that S3 was first & dominant early on: I'm stating it as a flat fact.

Yes, many RDBMS offerings have preceded Oracle, but none gained such wide adoption early on. Oracle has always had a strong sales team, so they've been able to grow their business rapidly. Good for them. SQL standardization has nothing to do with it either, because SQL is the user-facing language, not a wire protocol. What travels over the wire between an Oracle client and the server has about as much to do with SQL as the S3 API has to do with XML.

In your rush to defend AWS you seem to be falling into the same trap of immaturity as the people who have assumed that S3 will always remain the one true storage layer. It is not your fault, just a sign of the times. You're part of the generally apparent downward trend in level-headed, long-term thinking among the software cadre. For your next project, I recommend that you keep in mind that proper abstractions and clearly defined interfaces are crucial for maintainability of software over the long term. Good luck!

>It is not an accident that GCP and Azure storage service has a S3 API and not the other way around.

both look like third party projects rather than something by Google/msft. saying that "gcp/azure has a s3 API" would be like saying "Linux has a win32 API" because there's WINE.

S3Proxy is third-party, but Google's GCS S3 compatiblity is very much native. It was a selling point when they first launched it that you could just take boto and change the endpoint, add a header, and have it work.