I just don't want to be called to the office on a weekend anymore for this sort of BS.
Production incidents with MongoDB last year: 15 Production instances with redis, elasticsearch and mysql combined last year: 2 (and with much less severity)
Edit: just to add: I didn't pick Mongo, I was just the engineer called to clean that mess. I created enough of my own messes to not resent the person who made that shot for it. We are constantly on the verge of rewriting the MongoDB stuff since a database that small (~250GB) should really not have these many issues (In previous workplaces I ran ~10TB PostgreSQL deployments with much more complicated schemas and queries with far fewer issues). It's also expensive and support at Mongo Atlas hasn't been great (we should probably self host but I am not used to small databases being so problematic)
aw crap. oh well it probably doesn't matter for my small-ish application.
I suppose salespeople probably aren't into the nitty-gritty, but their tech people should have warned them about this. Maybe they were just trying to pull our collective leg, but I suppose that why I was at that meeting.
It was obviously an instant 'No'.
Even if we weren't - as a sales engineer on a large CMS/ECommerce platform with merchants running $150M+ in annual revenue, with an average client retention of seven years, and two decades of agency experience behind the decisions around building that platform, if you instantly said no just because of MongoDB, maybe you don't know as much about MongoDB as you think you do.
I came from a SQL background myself, and had reservations based on all the things I'd read about MongoDB as we decided to build a platform after doing things bespoke for two decades, but time has proven our architecture choices out. It's easy to be proud of something that works well.
My only experience with MongoDB is being "the engineer called to clean the mess". I'm sure you can effectively use MongoDB in production if you're knowledgable and careful, but most people aren't and they shouldn't have to know the detailed inner working to not create a mess.
It’s always the same
1. Newbie webdev (aren’t they all) uses MongoDB because it’s easy to use according to blogs and twitter
2. Somehow it makes it into production
3. A dozen experienced engineers spend years trying to keep it running
> Please respond to the strongest plausible interpretation of what someone says, not a weaker one that's easier to criticize. Assume good faith.
[1]: https://news.ycombinator.com/newsguidelines.html
In this case, the parent commenter probably meant that "newbie web developers" are likely to choose MongoDB. Of course, web developers have a range of experience, some new, some seasoned.
Regarding your comment, I am reminded of a pattern in online behavior over time: people seem to take offense more easily. Please take a look at https://www.psychologytoday.com/us/blog/how-do-life/201410/t... to understand what I mean.
Based on seeing how comments like this may get interpreted, as well as broader thinking about online communication, I think HN should consider a more nuanced system of comment feedback mechanisms.
I don't have a particular plan finalized, but I would like to see HN provide feedback on different aspects of the comment. Below are some important aspects:
To what degree does the reader / voter... *
1. agree/disagee with the comment?
2. find the comment relevant / irrelevant to the topic as a whole?
3. find the comment is situated in the correct / incorrect location in the thread? (e.g. responding to the parent comment or not) 4. find the comment interesting / uninteresting?
5. think the comment adds to a diversity of perspectives?
6. find the comment clear / unclear?
7. think the comment aligns / (does not align) with the HN Guidelines: https://news.ycombinator.com/newsguidelines.html
8. find the comment welcoming / offensive?
* When I write '/' above, I intend it to be a continuum; e.g. hot/cold means "in the continuum between hot and cold).
Additionally, being able to give feedback in a more granular fashion could be of use. For example, in my comment above, I would not be surprised if a significant number of people were bothered/offended by my commentary that people seem to be taking offense more easily. Some would call this ironic -- I wouldn't -- I think it gives more data to prove the point.
Motivations: my goal here is not to gain or lose karma -- I care very little about karma here, precisely because it is so muddled and varied from person to person -- as long as I have enough to participate fully. My goal is to learn and play a small part in fostering awareness and community, while hopefully to motivating others to reflect on their impact on the community here.
> "Newbie webdev (aren’t they all) uses MongoDB because it’s easy to use according to blogs and twitter"
I'm not sure if I completely missed the "aren't they all" part or if the comment got edited.*
So, clearly, in response to "aren't they all", the response by sophiebits above makes complete sense.
* This makes me wonder, is a comment locked for editing once a reply gets added?
The thing is that MySQL is older so it went through all of it earlier, but it still suffers from poor decisions from the past. This is contrasting with PostgreSQL, where correctness and reliability was #1 from the beginning. It started as an awfully slow database, but performance for improved and we now have correct, reliable and fast database.
Transaction ID wraparound: https://twitter.com/bcantrill/status/1110647418008133632
Incorrect use of fsync: https://news.ycombinator.com/item?id=19119991
If you were around back in the day you will remember the MySQL team claiming that no one needed transactions or referential integrity, that you should just do it yourself in the application...
ORMs and ActiveRecord in particular encourage, to some extent, the use of a RDBMS, even if they didn't get to take advantage of them well for a long time - for example, in RoR "has_one / has_many" for foreign-key relationship, .joins(:field_name) for, well, joins, and so on.
A big reason I called out RoR is that back in '04-05 I was railing against its default use of plural table names, and DHH on IRC recommended I shut up and just flip the configuration switch and turn off the feature, but of course when I did that all sorts of latent bugs were exposed.
RoR was the beginning of hipster "coding" and I therefore blame it for everything.
I'm wasn't previously familiar with the FriendFeed approach to database (ab)use. I paid about as much attention to it as I did to MySpace back in the day -- nearly zilch -- so its etc innards are doubly obscure to me.
> MySQL was the perfect database for people who had no respect for databases.
Very few database systems support online DDL, which unlike a transaction, does not require undo or rollback resources. Of course one must have a rollback procedure if something fails, but you need one for transactions too, just in case.
An online rollback is far lest costly than a transactional rollback, because and online rollback is just undoing what you did. Added a column you didn't want in one query? Remove it again in another, very quickly.
TokuDB (a mysql/mariadb storage engine) supported all DDL as an online operation. But percona killed it in favour of TokuMX, the MongoDB equivalent.
TokuMX has no upgrade path to wired tiger, only one major customer at Percona (I can't say who it is) and no engineers.
Any kind of DDL is tricky and requires users to RTFM for the intricacies of their chosen database. One size rarely fits all.
I really wish Percona would reconsider their decision to deprecate it.
After Percona took over TokuDB's creator TokuTek, they wasted so much of their development time and money on TokuMX (Percona's fractal tree-enabled MongoDB server) only to abandon it in 2017.
That money would have been better spent on TokuDB development to allow it to match the features present in InnoDB like generated columns, spatial indexing, fulltext Indexes and Galera.
TokuDB still has many users and MyRocks is just no substitute.
People rightfully joked about MySQL when they had the non-ACID engine.
Same for MongoDB. A database that loses data when properly used is a joke.
Yes, there are use cases out there for fast non-guaranteed writes. No, 99% of companies don’t have them.
Periodic snapshots of state held elsewhere. If you lose a write, you just get stale data until the next update.
Firm realtime work. If you lose a write, that sucks, but a slow write sucks just as much.
I could think of an instance where you’d like to log data, but the occasional datapoint being missing wouldn’t be terrible. Maybe something like a temperature monitor — you’d like to have a record of the temperature by the minute, but if a few records dropped out, you’d be able to guess the missing values from context. Something like the data monitoring equivalent of UDP vs TCP.
You then "checkpoint" when the game is over.
You might dissent that is not a "non-guaranted" write, because in fact the write did occur, but I simply want to allude to the concept of a "non-secured" write, in that it vanished without an fsync.
I work for a telco where we log large amounts of network requests using MongoDB.
Also would depend on how long time you think the application will be around. You're building an MVP to evaluate something? Just hack together whatever will work (then throw away). You're maintain software for a library/archive that will most likely stick around for a long time, even if they say it's just temporary? Do decisions that will help in the future, always.
We have a roadmap we need to meet and so far we have been trying to spill money on it rather than developers (paying mongo atlas) and adding features incrementally as Mongo gets them (like transactions).
If this wasn't a startup we would probably rewrite.
That's about a decade out of date at this point. MySQL/InnoDB is the standard table engine and corruption is exceedingly rare. As of 2014, when I last directly worked on MySQL prod systems, there was no practical difference from PostgreSQL in terms of transactional guarantees. That includes APIs like JDBC which we used for billions of transactions.
MariaDB [test]> create table test ( i int );
Query OK, 0 rows affected (0.06 sec)
MariaDB [test]> insert into test values (''), ('xxx');
Query OK, 2 row affected, 2 warning (0.01 sec)
MariaDB [test]> select * from test;
+------+
| i |
+------+
| 0 |
| 0 |
+------+
2 row in set (0.01 sec)
That's a new MariaDB 15.1 with the default settings I just installed the other day to test some WordPress stuff. I know there are warnings, and that you can configure this by adding STRICT_ALL_TABLES to SQL_MODE, but IMO it's a dangerous default.
This is also an issue with using MongoDB as a generic database: every time I've seen it used there were these kind of data integrity issues: sometimes minor, sometimes brining everything down. Jepsen reports aside, this alone should make people double-check if they really want or need MongoDB, because turns out that most of the time you don't really want this.
https://wiki.postgresql.org/wiki/Transactional_DDL_in_Postgr...
That MySQL autocommits is also even worse than just "doesn't support it."
A basic element of the relational model is that metadata is stored as relational data and that the same guarantees that apply to manipulating main data in the database apply to manipulating the schema metadata.
It's true that many real relational databases compromise on this element in various ways at times, but it is absolutely not the case that DDL “is supposed to be” non-transactional.
isn't that the point? you can use a session to do multi actions within that session.
withTransaction(async session => {
await Promise.all([someOp(sesson),
someOtherOp(session)]);
});
However, when the driver calls your function again it doesn't invalidate the `session` object - so previous calls to the same function can make updates to the database.
Let's say `someOp` does something that causes the transaction to retry and `someOtherOp` is doing something non-mongo-related in the meantime (like pulling a value from redis). Now `someOtherOp` reached the mongo part of its code and it is executing it happily with the same session object (so operations succeed although they really shouldn't)
The point of transactions like you said is to perform multiple operations atomically and for them to happen "exactly once or not at all". With Mongo in practice it is very easy to get "Once and some leftovers from a previous attempt".
What exactly would invalidating that session object do here? And what would the session object do after it was invalidated?
Because you wrap the DB operations inside a Promise.all, it means it will run them all BUT it will not revert them if one fails (it's not atomic, it just says that one has failed and you need to catch it), it will reject them but not revert them. (the CUD operation will already have changed the data) The problem I believe is the transaction is considering the Promise.all and not what's inside of it so it will run it again despite the fact that some have already succeeded earlier
I think you just have to resolve each of them outside a Promise.all. In your case because Promise.all has been rejected it will redo the transaction, therefor it will redo the one that have already worked in the first call.
I'm no expert but this is how I understand it.
Without promise.all, I think it can be replicated like this:
try{
await someOp(session);
catch { }
await someOtherOp(session);
They implemented a "disposer" incorrectly (I added this to several other libraries and made a Q&A about it here https://stackoverflow.com/questions/28915677/what-is-the-pro... )
Am I expected to justify being able to code in HN? I could go on about being a maintainer of the two most promise libraries in npm (bluebird and Q), being node core, organizing promises sessions for APIs in Node core and having over 1000 answers about promises in SO.
I generally find that sort of "non-technical chat" boring compared to the technical stuff and appeal to authority kind of lame.
The only other person attacked in this issue is aphyr, so I believe I am in very good company in this particular instance.
Just wondering, did you submit a bug report to them about this? If so, any response?
We reported it immediately at the highest severity and we pay for the highest tier or support - we tried to collaborate as soon as possible. It sort of went "over their head".
For example at #00636235 "how to avoid TransientTransactionError".
In mongo, you have a `withTransaction(fn)` helper that passes a session parameter. Mongo can call this function mutliple times with the same session object.
This means that if you have an async function with reference to a session and a transaction gets retried - you very often get "part of one attempt + some parts of another" committed.
We had to write a ton of logic around their poor implementation and I was shocked to see the code underneath.
It was just such a stark contrast to products that I worked with before that generally "just worked" like postgres, elasticsearch or redis. Even tools people joke about a lot like mysql never gave me this sort of data corruption.
Edit: I was kind of angry when writing this so I didn't provide a source and I'm a bit surprised this go so many upvotes without a source (I guess this community is more trusting than I assumed :] ). Anyway for good measure and to behave the way I'd like others to when making such accusations here is where they pass the same session object to the transacton https://github.com/mongodb/node-mongodb-native/blob/e5b762c6... (follow from withTransaction in that file) - I can add examples of code easily introducing the above mentioned bug if people are interested.