This sounds what big companies or a disorganized company would need. For an efficient enough company, a project like this needs just one or two dedicated engineers.

In fact, I can't imagine why this project needs a PM at all. The database is used by engineers and is built by engineers. Engineers should be their own PMs. It's like we need a PM for a programming language, but no, the compiler writer must be the language designer and must use the the language. Those who do not use a product or do not have in-depth knowledge in the the domain should not be the PM of the product.

Maybe for a research project or a hobby project, but not for a real, high performance database to be used in a business-critical application.

FTA:

"Databases are a nightmare to write, from Atomicity, Consistency, Isolation, and Durability (ACID) requirements to sharding to fault recovery to administration - everything is hard beyond belief."

>Engineers should be their own PMs.

For small projects, sure (your "one or two dedicated engineers"). But once you start tackling projects that require larger teams, or even teams of teams, you need someone to track and prioritize the work remaining and the work in progress (as well as the corresponding budgets for personnel, services, and other resources). Similar to the way a sole proprietor can do their own accounting, but a multi-million dollar business probably should have an accountant.

As an aside, I wonder if this might be a use case for a bitmap db engine like Featurebase (https://www.featurebase.com/).

The trick is that they didn't need a database that provides "Atomicity, Consistency, Isolation, and Durability (ACID)". By only implementing what they need they were able to keep the project small.

It's like people are scared of doing anything without making it into some huge multi hundred developer effort. They've written a super simple append only document store. It's not rocket science. It's not a general purpose arbitrary SQL database.

Maybe they really built only what they needed. If so, good on them, but ditch the clickbait title. On the other hand, a lot of people believe that they've built only what they need, only to discover in a week or a month or a year that they underestimated their own requirements. Then they gradually re-learn all the lessons that led to the development of real databases, to their eventual sorrow.

Everything we are teaching people to do is centered around solving the problems with already built services, preferably from one of the big 3 cloud vendors.

We are churning out “qualified” engineers who don’t know how anything works and we pay them to configure a few AWS services.

So that is at least 20k a month, for fairly cheap engineers.

What about when two different projects have two different requirements they need supported by the database. Which one is implemented first? What about if there is only engineering capacity to implement one?

I don’t think a database is the place for “just send a PR for adding your required feature and ping the team that owns it” kind of development. It requires research, planning, architecture review, testing, etc. It’s not a hobby project, it’s a critical tool for the business.

One of them. This is true whether you have a person named "PM" or not. It's just a matter of who picks.

> What about if there is only engineering capacity to implement one?

How does naming some guy "PM" solve the issue? The team just picks one of the features.

(Though to be fair, if I actually wanted to put this in prod it probably wouldn't take too long to fuzz it and fix the kinks.)

https://github.com/eatonphil/btree-rs

$60,000 per year in-house vs $1,200,000 per year aurora. No brainer really.

Note 'instances' eg plural, versus a singular EBS. There is some ambiguity here, I'm not sure where the 10x came from, but it seems plausible.

i saved hundreds of dollars per month by switching from an audi a4 to riding a home-built bike for my 1.5 mile commute to work

It really bugs me how everyone has drank the coolaide. Cloud is stupid expensive, but of course this is cloud vs cloud.

SANs add latency, who knew? What did surprise me during testing is that Aurora seems to have added latency even if the entire dataset could easily fit into buffers, with zero disk reads required.

If you want actually fast cloud DBaaS, the only way is with instances that have local NVMe for caching, like AWS ??gd.db instances.

Only if they planned on hiring someone just to develop this new database and if they switch to Aurora they’d let them go immediately. If the said developer was already costing them $250k to maintain and develop the application and work on top of Aurora cost seems like a good way to save $100k/year.

But at the same time, Aurora costs could also scale with usage. It may cost $120k one year, $180k next year, $500k the year after. If the database they have now is well designed after it's already built it may not need active development every year but adding a feature here and there. Also, switching back to Aurora could also be an opportunity cost "we should have written our own thing and could have saved millions ...".

Also they said their current volume is around 13k/second. They’ve built the new platform for 30k/sec per node. This should last them a long time with minimal maintenance.

If this db requires 1 full-time developer then the cost would immediately be not worth it (assuming salary + benefits > $120k/yr)

As you say, without details it’s hard to know if this was a good idea.

A large part of the value of popular platforms is precisely that they are not bespoke. You can hire engineers with MySQL/Postgres experience. You cannot hire engineers who already have experience with your bespoke systems.

That barely qualifies for the median mortgage in the US.

Now, median in the Seattle metro, or in San Francisco, sure. But 120k in, say, St Louis is still going to get you an intermediate dev, no problem, and they can afford a house by themselves too. There are 4 bedroom houses in my neighborhood for 300K.

They're operationally using a funny spelling of SQLite and I don't imagine anyone arguing that such a thing needs constant attention.

What do you need them for?

They mention 13.5k simultaneous connections. The US has 4.2 million tractors alone, just the US, just tractors. If they get 10% of those tractors on the network that's a 30x to their data storage needs. So multiply that across the entire planet, and all the use cases they hope to serve.

Investing time early on so that they can store 50x data-per-dollar is almost certainly time well spent.

S3 is relatively cheap.

They may or may not be doing other things depending on the company size and state.

You could drop the PM, engineers writing for engineers don’t need a PM.

You will likely hire similarly costed engineers to maintain the database stack anyways.

You basically hit all the talking points big cloud has brainwashed people into thinking into being true, but every day we see stories of a handful of engineers doing something we are told can’t be done and saving millions in cloud cost.

It’s so painful to watch. Software Engineering became a thing because you could hire a engineer solve your problem, and big businesses stepped in and told you that your problem was something else and gone out of its way to stifle innovation by settling industry standards on how to do things that only guarantee you use cloud services.

Any company that wants to own its destiny knows to stay away from lock-in.