Be careful with the language there....
I assume you meant that no-one wants to have to run and administer an RDBMS like Postgres, MSSQL, Oracle, etc - especially not things like backups, restoration, disaster-recovery, figure out storage, ensure high-availability, and maybe even things like user administration and index management: i.e. a DBA's grunt-work - and yes, I agree with you entirely.
...but using the other interpretation of "manage", people might think you meant that people don't want to know how to properly design a normalized schema and apply the business-domain, knowing how to clean-up data appropriately (i.e. the other half of a DBA's job), because we can either shove it all into a cloud vendor's proprietary (and horrendously expensive) non-relational database or data-lake solution (CosmosDB, etc) - or that we can outsource database modelling and implementation of DDL to an LLM - which I'll admit, is quite capable as-is for a first-draft for DB scaffolding and getting better all the time, but ultimately isn't able to take a proactive or autonomous approach to managing an org's databases; I don't want to seem like I'm advocating economic-protectionism of DBAs, but I get the impression LLM-based DBA-bots are being aggressively pitched to smaller-sized orgs that don't know any better, at a time when the technology is still very new, with potentially serious repercussions w.r.t. data-security and compliance (and ChatGPT still gives me T-SQL that uses Oracle functions...).
The self managed DB servers aren't cheap once you have to scale, typically have availability issues due to ACID rules typically limiting servers to only two AZ's per region etc...
It is actually adopting best practices for software maintainability and scalability that are getting rid of most DBA needs....because most databases use aren't doing batch processing, report generation, backups, job scheduling, or other needs that tend to be the right use cases for relational databases.
Concepts like encapsulation, persistence ignorance, and other methods of ensuring that the persistence layer is as loosely coupled as possible have proven to be cheaper, easier to test, and avoid the lock in you mention above.
As i have had a professional relationship with AWS/GCS here are a few anti-patterns from Microsoft that demonstrate this.
https://learn.microsoft.com/en-us/azure/architecture/antipat... https://learn.microsoft.com/en-us/azure/architecture/antipat... https://learn.microsoft.com/en-us/azure/architecture/antipat...
I made a good living as a consultant in the .com era helping companies throw money at their foreign key loving branching stored procedure systems that they couldn't even get to scale past the angel round some times.
And entire companies like AT&T wireless went under in a large part due to failed upgrades due to issues related to having logic in the DB.
DBs can be a good persistence layer when a system is simple and small. But really to work well within a cloud or distributed context they need to be treated as repositories unless there is a compelling technical reason to treat them differently.
'When all you have is a hammer' approach monolithic persistence layer what the primary blocker when I worked with cloud providers to help customers migrate. And they always were paranoid of moving things out of the database. And the funny thing is that most of them were trying to move to the cloud because vertical scaling on-prem DB servers became too expensive and didn't deliver the performance vendors promised.
Had companies followed say the design patterns of micro-services, where each service context has it's own DB, they probably could have stayed on-prem at lower costs.
But for the clients I worked with, who had failed to migrate on their own. Complex fragile schemas, triggers, sp's, etc... where almost always the main issue.
And we knew this even before the commercial internet existed.
> POSTQUEL allows users to save and query historical data and versions. By default, data in a relation is never deleted or updated. Conventional retrievals always access the current tuples in the relation. Historical data can be accessed by indicating the desired time when defining a tuple variable.
I looked this up during another thread which also has some other sources if you want to dig further: https://news.ycombinator.com/item?id=37956687
23 billion.
Thats the profit for AWS in 2023.
23 billon that AWS users pissed away.
AWS is NOT cheap. I have quite a few clients who, could save lots of money dumping their cloud provider and setting up their own racks, hiring SA's. I have others who, if they optimized any of their stack could cut their AWS bills by half.
Having survived the 2000 bubble the lessons are going to have to be re-learned. Everything has cost and value and managing that is going to become an engineering task in the near future.
Would these clients have been able to make the upfront investment to set up their own racks and hire / train the relevant staff? That's the sales pitch for AWS and other cloud providers, you can get started a lot faster, make money, only then look into optimizing.
Running your own servers is easy to underestimate. Running it with the reliability, backup and fallbacks that cloud providers offer is also easily overlooked. How much does it cost to set up a multi region cluster yourself?
That said, there are some real life examples of companies that moved away from AWS in favor of their own servers, like Dropbox which bootstrapped on AWS, then built their own eight years later (https://techcrunch.com/2017/09/15/why-dropbox-decided-to-dro...).
IME, companies often will first fall into the trap of minimum spend with AWS for larger discounts, such that even when you _do_ try to optimize things, it doesn’t matter. There is no greater disincentive than saving 5 digits per month, only to be told “thanks, but it doesn’t matter.”
Tangentially, I think the rise of cloud and SaaS to blame more than anything for the I-shaped engineer. Why bother learning anything about optimization when K8s will silently add another node to the cluster as needed? Why bother learning SQL when ORMs abound, and your Serverless DB will happily upsize to deal with your terrible schema? After all, you’re being told to ship, ship, ship, so there’s not really any time to learn those things anyway.
I simultaneously love and hate cloud providers, and am eagerly awaiting the slow but steady turn towards on-prem/colo. 37Signals has the right idea.
FYI: I have known people 8 years ago who already made money by optimizing people's AWS bills. "AWS expert", "AWS consultant" etc. are valid contractor titles.
I've realised that cloud costs are a "wave". (App is app/service/business)
- cheap to run as a small app
- costly to run as a bigger app, cheaper to self-host
- cheaper than running all infra for a much bigger app
- extremely costly compared to running all infra for a much-much bigger app
- probably on par with running your own infra (but then you're getting into tens/hundreds of millions of customers)
While I agree that running your own hardware can be _seriously_ cheaper, this is like saying that renting is a waste of money and everyone should purchase a home.
Capacity planning isn't easy for a lot of small companies. Neither is the engineering time to set up, manage, and troubleshoot the hardware. At a certain scale, these are no brainers to take into your own hands, but for a 15 person engineering team that's completely focused on the product, it would be pretty expensive in human cost to set up and manage your own hardware. Sometimes, it can be set-and-forget, but rarely will it be as set-and-forget as a cloud provider.
Is it more expensive overall? Yes. Is it "pissed away" money? In some cases, maybe. In many, no. The ease of running and reduced upfront capex is absolutely valuable.
Renting is a waste of money and everyone (or at least a lot more people than at present) should own one's own home, whether on their own or via e.g. a housing cooperative.
AWS might cost more but it provides a lot of benefits that are hard to assign a dollar value to.
Those things are typically factored into estimates for ownership v. rental costs, and even with those factored in, renting costs more than owning. That really shouldn't be terribly surprising, either; if landlords only broke even on their rentals, then there would be no reason to be in the rental business instead of the property management business.
If I know I'm going to be living in a city for two years, buying a home would likely be a bad idea. Sure we could talk about scenarios of renting it out and yada yada, but we're getting into corner cases in a comment section of a thread about AWS pricing :)
And bringing things back to AWS pricing: yeah, at first you might not know exactly what you need capacity-wise, but at some point (assuming you're actually measuring resource usage) you start to get a feel for what you need day-to-day and where things are trending. Much like with living arrangements, infrastructure hosting tends to have at least some pieces that you know you're gonna need long-term.
The more compelling reason to go with IaaS/PaaS is the convenience of not needing to run your own servers in your own rack - and it's that convenience that you're paying for, at a premium compared to hiring sysadmins. For a lot of businesses that's cost-effective, because they don't have/need enough infra to keep a sysadmin busy, but at some point that flips.
Are we that far removed from reality to forget that one small team of DB admins can handle literally thousands of instances?
We use AWS because collectively paying AWS 23billion is stupendously cheaper than 1m users AWS having a a whole infrastructure team.
How do you know it would have cost them less to hire their own teams for self-hosting?
Let the other startups spend x10-100 on their resume driven development and blaze through their runway. Common story actually.
But suppose you don't need it and RDS as-is covers all your needs, which it often does, why rage? This is a continuum.
If an early decision to punt and outsource it to the cloud was made it could always be reversed at an opportune moment. I don't think competent people see it as "the only way" just the most common and pragmatic option.
We use SQL Server's temporal-tables - and while they do introduce some restrictions (e.g. you can't mark a column as "ignored" for temporal changes, and only system-versioning (not application-versioning) is currently supported) the feature has overall been an absolute lifesaver for allowing us to do data-recovery from user-error on a per-record basis - all from within our application or directly in T-SQL; whereas previously we'd need to get do a full mount of a PITR backup which requires a lot of manual steps and can take at least an hour in most cases - and then hope that we got the exact point-in-time correct.
Even for its use as an auto-auditing system, it's great: we often get customer complaints about us "losing their data" or that we corrupted their records, when in-fact, we have the historical rows and can see that it was one of their own employees (or family-members...) mucking around. Previously we'd have cheerfly accepted the user's complaint and given them a refund or service-credit because it was too much hassle to dig-up the weeks-old HTTP logs or have anxieties about we having any non-transaction-safe OLTP code running causing corruption that we don't know about :S
Also double checking the docs because its been 5 years since I have reasonably touched SQL Server - no indexes on historical tables so it gets slow fast, Filetable doesn't work* /nvarchar(max) records bloat you hard core.
I find that a log record is much better as a non relational/document type record that is isomorphic to the record so I can schema on read if I need to, and if you want to JSON query that stuff it works fine too.
* but nobody uses it anymore
I use AWS / RDS - I’ve got to believe RDS is spinning money for AWS because of all the parts I want to be reliable this is way up there.
You might disagree with this statement of course but that is truly my experience (30+ years of working professionally in software development).
Try this: Next time a developer confidently suggests that you should use tech X, ask them that the pros/cons are compared with using tech Y/Z. Ask about performance/failover/latency/team skills required etc. Most developers in my experience completely fail to give an intelligent answer. They are just loudspeakers confidently copying the opinions of somebody else. Usually somebody who benefits financially from developers using tech X.
We're hoping to make it so that this becomes automatic with Postgres vs moving to specialized databases.
We recently open sourced the core tech that implements system versioned tables https://github.com/BemiHQ/bemi, check it out if interested :)
This really provokes a wild geek rage inside me. Not that it is untrue, but that this is the state of things. It's that cloud providers are now often seen as the only responsible way to operate infrastructure, and in the process we hand over so much control of our tooling. We let ourselves be convinced that simplicity was worth a 10x premium, and admitted we were not grown up enough to handle the tools we had made for ourselves.
Sure, I know it is more complex than that. But I also feel sad about it.