Epic had a post mortem blog post here that mentioned in passing they had stumped all the experts they could find to look at unsolveable issues they had with MongoDB. https://news.ycombinator.com/item?id=16340462 I kind of assumed the fix is going to be a rewrite with Postgres or MySQL.

- That's not what they said.

- You think people replace a MongoDB cluster by a single Posgres instance? You guys should really use HA, cluster in real life and stop reading reddit / HN and the hype behind PG, with 3.5M+ CCU no one would use an architecture with a single master / slave ( that's what pg is ).

MongoDB / MySQL have bad press by people that never used it in real life and just repeat what they read online.

I could tell you horror story about pg not have an official replication system until 2011 when pg 9.0 landed.

> I could tell you horror story about pg not have an official replication system until 2011 when pg 9.0 landed.

I could tell you a horror story that happened to me just few weeks ago, where MariaDB just corrupted data out of nowhere due to a bug[1]. This happened multiple times and costed us multiple hours of work (including service being down) each time it happened until we realized the issue wasn't hardware but a software bug.

If you ask me, I take PostgreSQL approach of not having a broken replication before 2011 than MySQLs still corrupting data. Data usually is the most valuable asset a company has.

[1] https://jira.mariadb.org/browse/MDEV-10977

> I could tell you horror story about pg not have an official replication system until 2011 when pg 9.0 landed.

And I would re-iterate that just because something isn't in mainline, doesn't mean it's not possible. Did you know that Pg didn't have native partitioning until Pg10? Somehow we managed to do partitioning before then.

I don't buy the argument that you need to ship broken features just to have them; Pg doesn't include it into base until it's a /good/ solution which is well engineered and has appropriate toggles. That is not a horror story.

> - You think people replace a MongoDB cluster by a single Posgres instance? You guys should really use HA, cluster in real life and stop reading reddit / HN and the hype behind PG, with 3.5M+ CCU no one would use an architecture with a single master / slave ( that's what pg is ).

I shipped a game which had similar CCUs (within the order of magnitude) and I can confirm that you can't do it with one postgresql machine, or.. actually you could but we chose to fsync() constantly to prevent corruption from ever happening and remove the RAID cache.. but you can shard on top of your database solution too.

You can cluster with postgres & mysql, but you have to implement the clustering / sharding logic yourself.

Nowadays I would just use redis & cassandra if you need something beyond a collection of postgres instances. Most projects do not.

For clustering MySQL, take a look at Vitess. It was developed at YouTube and recently adopted into CNCF:

https://www.cncf.io/blog/2018/02/05/cncf-host-vitess/

(Disclosure: I'm executive director of CNCF.)

I feel I need to quote the post mortem back at you so you point out where I mis read the quote.

"Our top focus right now is to ensure service availability. Our next steps are below: Identify and resolve the root cause of our DB performance issues. We’ve flown Mongo experts on-site to analyze our DB and usage, as well as provide real-time support during heavy load on weekends."

How does that disagree with my post?

You're implying they couldn't fix MongoDB or reached the limit which is false. In the current ( HN ) post they said they fixed it, I'm pretty sure they didn't have any experience in DBs in the first place hence why they asked for help.

Nowhere in the original post they mention issues related to MongoDB itself it was probably bad design on their side.

OK I should have been clearer about my interpretation - I read flying in experts as they flew in experts from MongoDB and stumped them so that had me thinking maybe this is not possible if they stumped them. Earlier in this thread one of the engineers from MongoDB says Epic solved the issue but had not updated the blog so I was wrong about that.

> mentioned in passing they had stumped all the experts they could find to look at unsolveable issues they had with MongoDB

That's not really what the article is saying, unless we interpret the following text differently.

"We’ve flown Mongo experts on-site to analyze our DB and usage, as well as provide real-time support during heavy load on weekends." -> "We have started to look into the problem together with experts" and not "Experts have tried and failed".

Yes, come over to the actually greener lawn of Postgres, it is amazing how much more powerful than MySQL it has become with recursive CTEs https://www.postgresql.org/docs/current/static/queries-with.... and some rather amazing JSON support https://www.postgresql.org/docs/current/static/functions-agg...

JSON has been great in MySQL for a few years now since 5.7, and recursive CTEs are coming in the next couple months with 8.0. I don't think you can make a wrong choice between the two these days, but choosing Mongo over either is almost always the wrong decision.

https://www.infoworld.com/article/3228154/sql/whats-new-in-m...

This is actually what I think the biggest power of JSONB is.

I can for example use jsonb_agg() and get a hierarchical response for 1:N joins. It returns JSON value even though neither of the columns contain JSON.

Previously in that scenario I would either need to make more than one query or get a response that has a lot of data repeated.

Another way of reading that post is that MongoDB is being used for some of the highest throughput concurrent workloads out there... and those are always hard to optimize. Doing a lift and shift for a "grass is greener" alternate solution is not a clear cut path to victory at all... but it's certainly a giant science project to contemplate.

Yes, I kind of wish MongoDB had come out and said what they are doing to help Epic Games here if this is something to address that issue or what the plan/thoughts are on the most newsworthy usage of MongoDB in a while.

I am one of the team of MongoDB engineers working with Epic on this issue, and I can assure you that the situation is under control and we have everything in place to scale this application to much higher numbers. However, we're not publishing details about our support cases, especially while they are in progress. That is something for Epic to decide, and I do assume they will eventually say in public just how well MongoDB is, in fact, performing for them.

Thanks for responding. I look forward to Epic updating their post mortem.

Not sure if you are aware but what you are asking for never happens.

It is not professional or appropriate for vendors to be revealing (a) that clients are having issues and need support and (b) the specific workings of technologies or processes within the client's business.

Funny to list MySQL here which preceded MongoDB with cheap/fast now, correctness later.

There's a whole 'nother generation of devs coming through who have never been burned by MongoDB though. Obviously they will be eventually, but by then another generation will come along to repeat the cycle.

I love deriding mongodb as much as the next dev that hasn't used it much; but I'll just note that while I'd still be hard pressed to prefer mysql over postgres - there was a long period where mysql was put to tasks it was ill suited for, especially prior to around version 4.x.

So while "hype first" might reap a deservedly abundant and bitter harvest of developer hatred - it doesn't preclude evolving into a genuinely useful product...

Both true, although I can completely understand why devs went with MySQL over PostgreSQL at that time. I remember that during the same time period that MySQL was drawing seemingly endless criticism for generally poor RDBMS behavior (3.x and 4.x), PostgreSQL was notorious for having poor performance due to insanely undersized default settings. Like out of the box it was sized to run with at most 10 MB of RAM or similar that was just unrealistic.

I also remember it also had a lot of quirks and missing features prior to v8. I assume it was leftover cruft from Ingres, but I remember PostgreSQL v6 and v7 being unreasonably complicated to get configured just because the defaults were so off reality.

One thing you can say about PostgreSQL, though, is that it's developers don't rest on their heels. Every major release packs in a ton of new features. They've gone from being fairly low or middling on the feature set to being pretty near the top. Even point releases have me saying, "Wow, that's really nice to have."

At the time oob postgres probably wasn't the (wrong) competition, sybase, Ms sql, Oracle was...

Or maybe, probably managed postgres. Esp. In the mysql 3.x days and earlier.

I used it about 3 years ago and my first thought was "How broken will multi-document ACID transactions be?"

I still want to like MongoDB, I still miss its style of query vs SQL, but I'd have a hard time advocating its use again...

Sometimes it's tempting to use it for projects that I know will remain small, but even then it's not worth the overhead of standing up a different DB when I have a perfectly good SQL server I can muddle through already.

Check out Rethink. It seems to be what you're after.

It's probably better to go with Cockroach or TiDB. RethinkDB has problems of its own.

We are in the process of evaluating CockroachDB vs Rethink internally and we've found CockroachDB to perform very poorly without obvious disk or CPU issues. I'm curious if you've seen different especially as it relates to Rethink.

I didn't do comparisons. But RethinkDB has straightforward issues like a slow QL implementation using a lot of CPU and a lot of disk space usage. Change feeds have a few scaling issues if you want a lot of them. I don't know that it has mysterious kinds of excessive resource usage. I'm a dev of RethinkDB, not an end user, so I might be seeing the worst side of it. I haven't used Cockroach or TiDB.

TiDB seems to have a pretty active community, judging by its repo (https://github.com/pingcap/tidb). Also, saw this thread on TiDB v MySQL recently that's pretty detailed (https://www.quora.com/How-does-TiDB-compare-with-MySQL). Looks like a good option worth trying out.

Can you clarify on those problems?

Performance, mostly. Too much CPU and disk usage. Change feeds don't scale well.

What problems (outside I guess the current status of the project as a whole)?

I work in a Danish muniplicity. Traditionally we've build everything on SQL because it's the world we function in, but we adopted the MEAN stack as a proof of concept a few years back and Mongo has been growing ever since.

It does require building and maintaining schemas in a different manner, but when you do that, it's pretty great to work with, especially when we're doing design driven development that consists of a lot of prototyping.

I'm a fan, but I'm a manager on business development and digitisation, so I may be a little sheltered from whatever annoyances it may cause in operations.

I work on a mongodb installation in a danish municipality. From the technical side mongodb has been great to work with.

I am curious why a municipality needs custom software. I mean, the scandinavian countries had standardised paper forms for most municipal tasks (population register, ledgers etc) already in the 17-18th century, and those were used nationwide, or at least throughout a single province. Why can't the same be done with software?

Well there are 98 municipalities and 98 ways to operate in a thousand different ways.

I’ve worked on quite a few multi-municipalitiy open source projects, like handling employee refunds on driving.

Basically I drive x kilometers for a meeting, I get paid x and the taxman gets the report. Simple stuff.

Well in the 6 parties involved there were 6 ways to interpret tax laws, 4 different agreements with unions on what rates to pay, 3 different payment systems with 3 very different ways of taking the reported data from a flat file to a rest interface, at least one political decision to overrule tax laws for a certain set of employeees and several different ideas on how to host it and so single sign on, oh and 4 different ways to obtain employee data.

That’s for a simple system with basically 1 function. We have more than 350 it-systems.

Another example is in automation. We have a scanner software and we have an archiving system. They both have APIs but the APIs speak very differ languages. This meant that our local scanner people were tasked with distribution after they scanned things, a task taking several hours each week because putting files into many different areas of an archive sucks. What we did was ask the scanning company to build a QR reader into their software, and then we made a piece of software that put the archive recipient addresses into QR codes. We also made a MOX agent, that accepta the output of our scanning software and loads it into the archive through the API. So now the process of distributing is automated.

You can certainly run a municipality without developers, using standard software and outside hires, it’s just really expensive.

Would it be fair to say that the political entity one step above the municipalities (whatever that is in Denmark) are not doing their job? I mean not doing their job on standardising things that can be in common between the municipalities. Some things will of course have to differ, but a lot of stuff likely differ just because not-invented-here. It sounds like the legislative environment is too complex, and that you have to work around it with a ton of software. Could it even be the case that computer systems have somewhat removed the incentive for the administration to rationalise the various systems? With just manual labor and typewriters all of that would have been very expensive, but with a server hall and a medium-size IT-team it kind of works out. Perhaps digitalisation only having come half-way is a factor - you mention scanning, but by now the so called "paper free office" that was a buzzword in the 1990s should be here already. Or is it perhaps just another sign that the IT industry overall is still very immature and this will sort itself out with time?

I think it's too complicated to blame anyone really. I mean, we working on standardising as much as possible, but it's often impossible because business practices are just so different. Often big standard products fall extremely short, or end up in complete failures because you can't jam people into boxes on an enterprise scale especially not when the people who build the systems have next to no domain knowledge and the people who write contracts have no technical knowledge. :)

I guess our government should work on writing laws that are more friendly to digitisation and stop expecting IT to fix business practices that don't really make sense in the first place. There has been a genuine movement toward that, but it's slow because none of our top politicians or bureaucrats are from technical fields, and they operate on such a high strategic level that they're often rather far from the daily challenges in a daycare institution.

Local political leadership and bureaucracy could certainly do more to focus on corporation, standardisation and digital transformation, and they actually do, but political views differ and they change every 4 years, and the truth is that there just isn't any voter interest in IT unless it goes wrong.

We're trying to build national standards, we've had a set of architectural standards called Rammearkitekturen for a few yers now, but getting them implemented is slow. For one they're made by muniplicities and our structure of government is split in three. Muniplicities, Counties and the State and each branch has it's own ideas, leading to bureaucracy and political differences. Some want us to use EU standards, others want us to build our own, and even if we decided, there are different sets of EU standards as well as different sets of Danish standards.

I personally think the best we can do is try to use whatever national standards are in favour, and build smaller applications on them, with open API's, and run everything as SaaS in infrastructures such as AWS or Azure. I also think we should do a lot more work on business development, modifying business practices before we throw IT at something.

But it's complicated and it's on a giant scale where even minor changes take years to implement

I used it quite happily circa 2009 and at a different company in 2014. In both cases it was being added to systems that already had mature functionality built atop a RDBMS. In the first case it was used to store events that had started to overwhelm the main RDBMS with write volume. (Originally a system with one database as the monolithic data store.) Probably Kafka would have been even better for this use case, had Kafka been available at the time. But MongoDB did the job very well. I did a prototype in Cassandra too before settling on MongoDB, but MongoDB had much better docs, drivers, and single-node read performance at the time.

The second time I used MongoDB to automatically track templated email bodies that were being delivered through a third party mail platform. We had dozens of recurring templates and many more one-off templates for different curated campaigns. If somebody complained that a link or image or token was wrong in their email, we wanted to be able to look back at the history to see if the problem was in the template data or potentially a client issue on their side. Most of the queries were ad-hoc and not very performance-sensitive. This was where a flexible JSON document format came in handy. Modern Postgres would have worked well for it too, but that wasn't available in the company at the time. With MongoDB I got good flexibility, adequate speed, and I avoided reinventing wheels by not trying to shoehorn the data into another MySQL table. I was able to solve a customer support pain point in less than a week and the system has worked well for nearly 4 years now.

I'd be really frustrated if I had to use MongoDB as my only data store. I would guess that much of the hate for it comes from people who were forced into that position, or maybe from people who didn't take its documented limitations seriously enough before productionizing its use.

I don't know much about MongoDB. I am mostly a client-side developer after all.

But every time I see a team transitioning from Mongo to something else, they transition to a relational database. May be their problem is not with MongoDB, but that their data is relational after all?

Personally, I'd take a relational db over NoSQL for most of my needs, but all these stories don't really say anything about how Mongo compares to other NoSQL databases.

That's because nearly all data is relational. At first it seems like you don't need a relational database, in fact NoSQL seems easier to use at first.

As your data grows though you realize that your application become more and more complex. A single query might translate to multiple queries to the database, you need to handle scenarios where fields might not exist etc.

With relational data you might have more work at front, but then the database solves many of the problems for you.

As another person said, when you're using databases like MongoDB you're going back in time and reliving the history, because databases in the past looked a lot like that before Codd invented the relational model, for example [1].

Also the whole NoSQL thing seems to be cyclical, we had XML databases in early 2000s[2].

[1] https://en.wikipedia.org/wiki/Hierarchical_database_model

[2] https://en.wikipedia.org/wiki/XML_database

Or maybe it's useful to get started off with a low overhead, easy to implement DB like Mongo and then as you grow larger, spin off uses that it doesn't serve well to other more specialized and complicated DBs?

One of the biggest problem with relational DBs is that once you decide on a schema, if it's the wrong one, you're gonna be in a lot of pain. Which makes a NoSQL DB a great fit for an early stage product where you are still figuring out what your product needs to do and contain. Once you have some more experience with it, and have a better understanding of your data, it's far easier to build the correct relationships.

> Or maybe it's useful to get started off with a low overhead, easy to implement DB like Mongo and then as you grow larger, spin off uses that it doesn't serve well to other more specialized and complicated DBs?

Not really, converting to relational data is quite a bit of work.

Actually the reverse is the correct approach. You start with normalized data, when there's a bottleneck you start denormalizing it, if that's still not enough you move /subset/ of data to NoSQL database.

> One of the biggest problem with relational DBs is that once you decide on a schema, if it's the wrong one, you're gonna be in a lot of pain.

Not really from my experience all migrations were done through SQL. Also if multiple people (who understand relational databases) come with a schema they pretty much will arrive to the same normalized result.

> every time I see a team transitioning from Mongo to something else, they transition to a relational database

They are repeating the discoveries that people made in the 1970s about storing data in flat files vs. relational models.

Know history or be doomed to repeat it and all that.

Yep this is me. I would require some pretty amazing reasons to even consider using Mongo again. Especially now all other relational databases i trust support json column types.

> However, I do not know any developers who, after living through the "hype first, features later" strategy, have been left with a positive enough opinion of MongoDB to ever want to use it again.

A new crop of developers is, always, just an year away though. I feel future adoption depends a lot on how well-suited the tools are for younger devs. That's where MongoDB found the initial audience!