Well, we're kind of comparing MongoDB of ~2011 (when The Guardian started using them) with Postgres of today.

One major change is that in 2010 you couldn't always run your whole DB in RAM, so there were some real performance benefits with MongoDB.

Another difference is that MongoDB was early with great JSON-support. Something that Postgres has since gained.

I think there are pros and cons with both. If I'd chose one today, for most tasks I'd probably chose Postgres.

That said, to understand why people made those choices, you need to 'teleport back' to that time and compare them in that year (given the tradeoffs at that time).

If somebody is teleporting back then, the fact Mongo had a _global_ Mongo instance level lock for writes should be more than enough for people to run screaming.

I worked with databases that only had table level locks(not row level) and there were more than enough occasions I cursed the creators.

Instance level(& indeed DB level) is insanity unless your DB is a read only DB.

Not saying you're wrong, but a news site and CMS has different database requirements than many other products.

They have very few write actions; in the Guardian's case hundreds of authors publishing a few articles a day, versus millions of users reading data. Writes would be sporadic and batched.

News products also changes more rapidly than you might expect. A modern news team may be writing and shipping code to better cover a breaking news event. I can imagine why a document store would be appealing to them.

I don't think Mongo was used much by people who would know why instance level lock is bad, or even what it is. And if someone who knew came later, it was too late - Mongo's proprietary querying language is a vendor lock-in.

Oh! From The Guardian write up I had the impression they were using it until mid 2018..

It is not easy to switch, so until the pain is too big, one doesn't do it.

For one thing, a good chunk of what happens in the industry isn't particularly well described by the term "engineering," which in my mind describes someone who is well-trained and experience-versed in relevant scientific models for the application domain and the dynamics of relevant "materials" and their interaction that can be composed into a solution. It might be the term "craft" better describes a lot of software development activity.

For another, it may be everyone is susceptible to the influence of trends. Even engineers. And the more complex the details of a subdomain of software engineering are, the bigger the tradeoff between becoming someone who can make a true engineering assessment in that niche and developing expertise elsewhere...

It's also largely a fact that we're paid better for following hypes, at least in the UK. Thus a smart "engineer" who also cares about their bank balance pretty much has to use everything new to stay relevant.

Further, most of the use cases are fairly shallow, so it doesn't actually matter that much, and you'll be paid more to paper over the cracks.

We need to change the market, but it doesn't like the strong and stable engineers are getting lead roles to pay more strong and stable engineers to build boring tech, despite that actually being better for most businesses.

Probably for the same reason people have a build process to load in 100s of KBs of Javascript in order to render an <h1> tag, and the compile it all down to HTML and declare static sites are the wave of the future.

You're going to need some supporting arguments if you're actually trying to say that every document database is not worth using.

Sometimes the choice isn't in the hands of the person most qualified to make that decision.

They are not overhyped, definitely not as much as old school RDBMS stuff. It's just most engineers can't make good database choices no matter what database they choose. Or more generally they can't make good infrastructure choices, as those are out of their competence and are mostly about things like operations and distributed systems, that take a long time and a lot of experience to get to a level of good decisions.

You cannot be old-school and hype. That's the reason why it is old school. And the people who can't make good database design choices are exactly the kind of people who should be using SQL. Postgres knows how to optimize and plan queries efficiently based on the actual distributions of values in your dataset. These poor choosers should be doing that... by hand?

https://www.postgresql.org/docs/11/planner-optimizer.html

hipsters are allegedly the cool old-schoolers.

the whole promise of mongo is/was distributed (HA+LB), which was all the rage back then, when AWS AZs dropped like flies every few weeks and scaling was seen as the problem. go fast, break things was the mantra.

and it's still not trivial to do pgsql maintenance without downtime, whereas in a clustered/distributed "solution", you can enjoy certain additional freedoms. this includes the freedom to shoot yourself in the foot (data inconsistency, but you had to fight a bit for that state by promoting a non latest slave to master).

> the whole promise of mongo is/was distributed (HA+LB)

Yes. It is so easy to bring up a mongo cluster and feel like you have HA. Don't worry it won't be proven wrong until you have writes during your cluster degradation and are unlucky.

it works. 4.0 gives you transactions.

it's pretty great.

and, most importantly, as far as I know the default replica set config is safe. (durable, consistent, atomic) it handles a lot of failover scenarios for you automatically. and you have to manually intervene to get into a bad state - which might be okay for that particular business case (better than full downtime)

This is what I'm talking about. You think that some feature that makes performance unpredictable is that important. While it's the last thing you should care about.

What people should care about is the integrity and consistency of the data they're storing. RDBMS do an extremely good job of generalising this problem with reasonable performance, which suits the majority of real world problems. The ideas behind it are understood by most of the industry.

It should be the default choice (or plain filesystem storage), unless you have a specific requirement for something different. In the latter case, this should be people who are well informed about the database choices they're making.

Unpredictable performance would mean, you can't predict how long a specific query will take. This is the opposite of how postgres works. If a query is slow, it's predictably slow if the data doesn't change. So what you must mean is, when the data changes, it doesn't behave like you expect.

So again, how do these poor choosers make better decisions in the face of changing data when they have to write all the algorithms by hand, and understand how to scale the data correctly? Seems to me that pretty soon they'd just be writing a very crappy database on top of some keystore.

The only way for query performance to be truly predictable is for your data to be static. If that's not your situation then a query optimizer is incredibly helpful because it means that you don't have to rewrite your queries every time you add an index.

It also means that as your data grows or shrinks, the optimizer will notice and change the plan accordingly if it makes sense.

That means that 18 months after you implemented a feature you've long since forgotten about, you don't have to come back and figure out what the new plan should be. And that's huuuuge.

You can find accounts of the db changing the plan from a good one to a bad one, but I'll go out on not that far of a limb and say that those are < 1% of the cases. Nobody complains about the queries they didn't have to come back and change. And the better the optimizer, the better that trade off will be.

> Why do ostensibly smart engineers keep falling for this stuff?

Review driven design is likely a large contributer. If you know you will be looking for a new job in 2 years, you better get some experience in $latestTrend instead of $provenTechnology.

perhaps they don't agree with your thesis.