I used https://calculator.aws/#/estimate and my number is over an order of magnitude less than $8000 for 2 instances with SSDs, 4GB RAM, and 2 cores.
Even just the case of sitemap access alone can very well read half a database's tables and many of each table's columns in the act. That being done dozens of times a week between various search vendors and yeah your egress from just SEO can exceed the size of your database alone. Just depends on the nature of your data.
Then consider paging views, infinite scroll.
Then toss in something like Algolia or other third-party tools ingesting your data.
But it really boils down to your application. You might have NONE of these and egress is just crumbs.
Unless your database is publicly accessible (which it really should not be), its egress should be zero. What your /application/ returns to the public internet, is a different conversation.
Let's say that you pay a lot. Like you expend $20.000 USD doing this.
But we forget the cost power of software.
Your monthly savings extend to the infinity.
Your dev is now freer to do other things that are precluded in a more complex scenario ("you save pennies here, but expend pounds elsewhere"), because complexity DEMANDS MORE ATTENTION.
This is something I have observed all my life. I have never regretted to optime]ize for operational costs even if this means I "waste" a lot of time for this. I work in a more cutthroat environment than most startups (ERPs), where my time to deploy to production something is measured in hours, and is an environment that is filled by the graveyard of my competitors that have drink all the kool-aids.
Complex operational environments ALWAYS have turned worse/more complex than do the simplest things.
P.D: I think is similar to spending time in refactoring, cleanups, testing, security audits, training, and even resting.
All of that sounds like "costs". But you are doing this stuff alongside, just need to budget a few hours a week on this and the profits come and come.
Never regret using time on this stuff.
NEED to be smart on this? Sure. But is pure muscle and after a while is great.
We're not talking about K8s or sharding. We're talking about adding more cores and RAM to the single database node that they're already using.
Imagine how much easier the backend dev's life would be if they could just write SELECT FROM JOIN WHERE like everyone else, and not have to pore over query plans and design/implement/test a bunch of custom joining routines. This path sounds like a high-effort artisanal approach to something where it doesn't seem warranted. It's penny-wise and pound-foolish in exactly the way you (very nicely) described.
That's not necessarily true of the business.
I am begging people to learn SQL; it is not a hard language, and it hasn't changed in decades.
Once you have the basics of SQL down (like how to write a query), you want to understand what the database actually does with your queries, so you can predict what the query plan "ought to be"/what you want, and you can read the query plans to try to understand why you did or didn't get the plan you expect.
https://use-the-index-luke.com/
And then once you understand the fundamentals, your database's manual is the place to go. Especially different index types (including their internals!) and what kinds of queries they support. The above link goes into depth about b-trees, but there are other types, and you can use b-trees as a reference point to contrast with others.
I was fortunate to have a Database Basics course in the university that was 80% database schema design and 20% SQL queries and indexes. Reasoning for such a split was that fixing a bad database schema is much harder than fixing bad queries.
The StackOverflow folks famously take their DB design and performance very seriously, and now they do not need an army of overpriced "architects" to plug the performance holes with more insane and expensive complexity.
https://stackexchange.com/performance
Coincidentally I just wrote up a post about the importance of minding your database skills:
https://renegadeotter.com/2023/11/12/your-database-skills-ar...
Having a simple monolithic database is not optimizing your DB engineering by any means.
edit: and inventing a better ORM doesn't count either.
I've seen far too experienced developers with no basic understanding of how relational databases operate. And it's gotten worse over the years - as they can keep throwing hardware at the problem. Performance optimization has become a lost art. This is as much a problem of negligence as much as it is of education.
I won't even touch on proper design, which requires someone with significant database expertise from the get-go ( which most startups may not have).
It's really not very hard, basically add indexes wherever lookups are done but don't go overboard, especially with tables that see a lot of inserts. The DB will tell you what it's doing if you ask, it's all easily investigated. If your joins resist optimisation your data structure is probably off.
> I won't even touch on proper design, which requires someone with significant database expertise from the get-go
In more extreme cases this is true, but a few basic principles and an understanding of databases will get you 95% of the way there and is quite intuitive. Something as basic as "don't copy data around, each individual entity exists in one place, and use its key to refer to it otherwise" will help enormously.
It's really not but you'd be surprised how many people don't know the basics of how indexes work. I've had people tell me that their queries couldn't be optimized further because "they had already added an index for every column of the table".
It doesn't help that SQL, by design, hides the actual algorithm doing the data access from the users while simultaneously relying on them to add indexes to achieve performance, which is in my humble opinion the worst mistake of SQL.
I want them to say “the heck are you talking about, because that’s not how anything works?”… it does trip some people up though that just legitimately don’t understand what’s happening.
(I only skimmed -- far too much "Joan explains that" to signal ratio.)
Interesting read on a project implementing this for Postgres: https://pganalyze.com/blog/automatic-indexing-system-postgre...
It will tell you _what_ it's doing, but not _why_ it's doing it. Unraveling the mysteries of the query planner is a dark art that is still firmly in the realm of graybeards.
Adding indexes can increase I/O time (more writes), increase DB file sizes (more stuff to store) etc, which are not inconsequential. Sometimes it's also as simple as moving indexes to a tablespace stored on an SSD.
Indeed! One place I worked at, there was something like 10x (forget exactly) gains to be had simply by editing the MySQL config to match the workload. Seems like nobody bothers anymore. There is so much money to be saved by reducing excess operating costs.
I think you have a mental model in which these people are spending weeks and months optimizing a system they should just upgrade while the real tasks languish, but in my experience, probably all they are doing is just paying a bit of attention every so often and simply doing things that don't require 32 cores and terabytes of RAM. A lot of things don't need that power.
20 years ago you had a lot more real variance in power. If your 100MHz server couldn't handle it, maybe your 400MHz server could, and the gap between those two systems is probably even bigger than the numbers imply because that 400MHz was probably equipped with better everything else (RAM, disk, cache, better performance/cycle for the CPU) to go with it. Nowadays, with CPU speeds having stalled, it's easy to go wide and service lots of diverse queries at a time, but if you have a particular query that your lowest-end DB node is struggling with, there's a pretty decent chance that just pushing the button to go to a bigger node will have a distinctly sub-linear result on performance improvement. I've certainly seen this from a number of teams in a number of places. You can push a button to get more CPUs but you can't push a button to get much better CPUs. Some better, yes, but not like it used to be.
If you just try a little, modern commodity systems can do a lot. Even with only 4 cores and 4GB. Just try a little. Yes, there are absolutely tasks that that doesn't work with; if you've got 10,000 hours of video to re-encode that system isn't going to cut it. But a lot of developers are really bad at understanding how long things should take and don't realize that the thing that takes two minutes during CI really should take something more like 50ms because https://accidentallyquadratic.tumblr.com/ and somebody really ought to look into that rather than just bump the CI node size again hoping it solves the problem.
(And for Pete's sake, when bumping the instance size does happen to not solve the problem... turn it back down again! Despite what I just wrote I do understand there's a time and a place to spray money at the problem, but when spraying money at the problem doesn't even help, STOP.)
The sad thing is that the things they discuss like monitoring and optimizing slow queries have always, until recently, been baseline competence.
It's only recently that the universal answer to something being too slow is to simply spin up another dozen, or heck make it two dozen, cloud instances to compensate.
> Even if their DB infra was only half as efficient as it is today, their costs would increase by a whopping $8000/yr.
I think the interesting comparison is not so much the same team doing things slightly less efficiently, but comparing to the average status quo in the industry.
So many companies are spending $100K+/month (month not year) on AWS costs to run very tiny minimal loads due to the attitude that efficiency never matters, just spin up more instances. Pretty soon you're talking real money. And by the time the CFO starts screaming, it's very difficult to go back to a more sane infrastructure.
Install Percona Monitoring & Management. Clickhouse + Grafana all in one, dashboards ready to go, and free. Not hard.
> optimization, removing joins, removing aggregations?
How is this not part of the job already? Too many people seem to have it in their head that the database is an inconvenience that gets in the way of their application. It is the heart and soul of every application, because all applications boil down to CRUD. You should learn SQL, full stop. Relation algebra is not that difficult. The time people spending learning the newest stupid JS framework could instead be spent learning a language that hasn't changed in decades. It will serve you well.
Are you sure about this estimation? Because Digital Ocean offers 2vCPU + 8GB machines for $25 / month, which is more than 10 times cheaper.
How much do they pay an average software developer? How many hours have developers spent on query monitoring, optimization, removing joins, removing aggregations? What is the ongoing cost of learning and applying these techniques to all new code written at the company? Even with the most conservative estimates I'm confident that it costs them several orders of magnitude more in developer productivity than they are saving on hardware.
Optimization is good, but every company needs to recognize where they are throwing away dollars to save cents. There is an optimal level of inefficiency that companies want to have.