News for hackers. Stuff that matters.
Re:MongoDB gets support for multi-document ACID transactions by overcast on Thursday February 15, 2018 @10:18AM (#16384659)
Next up will be SQL compliance, and we'll be back to a relational database. I'm curious as to what the impact to speed will be, and what the use cases for these types of databases is now that the major SQL players support JSON.
Re:MongoDB gets support for multi-document ACID transactions by drmirror on Thursday February 15, 2018 @10:43AM (#16384884)
I beg to differ. (Disclosure: I work for MongoDB.) Using JSON as your data model, rather than relational tables, lets you build different applications that don't need multi-document transactions as often, because the data is already together in a single document. But when you do need multi-document transactions (a small percentage of applications do, and only few use cases inside those applications), they are now available. There is no speed impact on cases when you don't use them. And most of the time, you shouldn't use them, otherwise you wouldn't be capitalizing on the advantages of JSON. I think that's a game changer, but then again: I do work for MongoDB.
Re:MongoDB gets support for multi-document ACID transactions by jeremiep on Thursday February 15, 2018 @11:57AM (#16385541)
Its usually only after a while you realize almost every piece of meaningful data is relational. It just didn't look that way when the project started. But now you're committed on the wrong database and its very costly to switch back to SQL.

Literally every project I saw using MongoDB ended up going back to SQL within the first 2 years after realizing the data is indeed very much relational and theres no clean way to model it using documents.

You always end up with either tons of duplication across documents, which is hell to maintain, or tons of multi-document queries with hacks to look ACID, which is also hell to maintain.

Sure Mongo makes it easy to prototype applications, but it makes it very complex to build robust and maintainable software. Its especially bad if you think your data isn't relational, because it almost certainly is.

Disclaimer: I believe Datomic to be the game changing database; because it values simplicity and composition and these attributes drive the entire design.

Re:MongoDB gets support for multi-document ACID transactions by fnayr on Thursday February 15, 2018 @01:43PM (#16386417)
Video games storing player data are a great example of nonrelational data. I'm intending on writing a blog post after I finish my game detailing the structure of data I store and why it was so perfect for mongodb.
Re:MongoDB gets support for multi-document ACID transactions by mustardo on Thursday February 15, 2018 @06:28PM (#16388698)
On the surface it sounds like you might have a case for Mongo, lookout for scenarios like...

* Trading in game items between two users (needs multi document atomic locks if you don't want duplicate or lost items) assuming your "schema" is a document per user

* You want to rename or restructure an attribute in the future, with no schema it's not possible change migrate data easily without writing ad hoc code (maybe you can use third party tools) or changing queries to expect data in multiple "schemas" which quickly gets painful

Good Luck!

Re:MongoDB gets support for multi-document ACID transactions by mygo on Thursday February 15, 2018 @06:46PM (#16388803)
> You want to rename or restructure an attribute in the future, with no schema it's not possible change migrate data easily without writing ad hoc code (maybe you can use third party tools) or changing queries to expect data in multiple "schemas" which quickly gets painful

You can have schemas with MongoDB. There are various libraries to facilitate database design by schema specification.

Also renaming or restructuring your data is not necessarily an easy task with SQL. The nature of a database dictates that how good it works for your application depends on up to how well thought-out your schema is. Having to change your schema around is tasking. One of the reported advantages of document stores when they were becoming trendy was that it was easy to change your schema since your schema is essentially determined and regulated at the application layer.

Also MongoDB has ACIDic transactions now (freaking finally) so if it’s as-advertised then I feel like half of your argument is not really a strong one any more.

Re:MongoDB gets support for multi-document ACID transactions by chris_wot on Thursday February 15, 2018 @10:11PM (#16389568)
If you need a schema, then go with an RDBMS.
Re:MongoDB gets support for multi-document ACID transactions by fnayr on Thursday February 15, 2018 @06:34PM (#16388747)
Yes players can sell items to other players, so that's the one place so far I've needed to worry about atomicity, but even mongodb docs give examples with how to deal with something like that: https://docs.mongodb.com/manual/tutorial/perform-two-phase-c...

So yes, it's annoying for a very small % of what I'm doing, but 99% of my updates/writes are within a single document, so I find it very nice for development.

Re:MongoDB gets support for multi-document ACID transactions by drmirror on Thursday February 15, 2018 @06:44PM (#16388795)
About the first use case, well it turns out MongoDB just introduced multi-document transactions today.
Re:MongoDB gets support for multi-document ACID transactions by mustardo on Thursday February 15, 2018 @07:05PM (#16388894)
If only someone had posted that information to HN
Re:MongoDB gets support for multi-document ACID transactions by javiermaestro on Thursday February 15, 2018 @03:10PM (#16387153)
Please do, because I'm interested in seeing how non-relational it is... I truly believe it has to be otherwise, so I'd love to read your findings!
Re:MongoDB gets support for multi-document ACID transactions by imtringued on Thursday February 15, 2018 @06:17PM (#16388641)
You still need ACID transactions over multiple entries even if your data is nonrelational otherwise there is the potential for item and money duping bugs.

A simple example would be a marketplace.

Player buys item X with Y gold from another player.

1. Server checks that item X exists.

2. Server checks that the player has at least Y gold.

3. Server removes the gold from the player

4. Server gives gold to the seller.

5. Server removes item from marketplace.

6. Server adds item to inventory.

What if someone maliciously crafts two requests in a way that step 2 of the second request happens before step 3 of the first request? The money is deducted properly but the account can now have a negative balance and there are now two instances of the item.

Re:MongoDB gets support for multi-document ACID transactions by nemo44x on Thursday February 15, 2018 @03:24PM (#16387279)
Most meaningful data is in fact not relational. When you consider machine generated data, logs, metrics, network event data and all other types of data like this you find there are not relations in it.

This data is meaningful because it allows to analyze what's going on over massive systems, detect when problems will happen, find bottle necks in applications and infrastructure, among many other use cases.

Application domain data tends to be relational I'd agree. But in general, this makes up a very small percentage of meaningful data in the world.

Re:MongoDB gets support for multi-document ACID transactions by aeorgnoieang on Thursday February 15, 2018 @04:04PM (#16387615)
> When you consider machine generated data, logs, metrics, network event data and all other types of data like this you find there are not relations in it.

I find that hard to believe. Maybe not that the raw data isn't already relational, but that there are no relations real or implied.

If logs contain info about 'things' and any of those things can be considered to be the 'same thing' for multiple entries, then there's a relation right there – entry to thing.

And even metrics and network event data I'd expect to be full of cryptic IDs that reference some 'thing', i.e. a typical 'code' for which it's really nice to have a table with at least a friendly description.

Admittedly some of this data – or maybe even most of this data – isn't very 'deeply relational', but it definitely seems that claiming that "there are [no] relations in it" isn't strictly true.

Re:MongoDB gets support for multi-document ACID transactions by nemo44x on Thursday February 15, 2018 @06:18PM (#16388644)
Well, it's all a point of view really. The data is in the form of an "event". An occurrence of a fact which occurred at a particular time and has data associated with it. So therefore, a "relation" as constructed in a relational database isn't appropriate. You aren't truly denormalizing the data when repeating ID's or tags or labels in this type of data. This is because, at that time that was in the fact the associated ID, tag, label, etc. It would make the stored event false if a field associated with it were to be changed as at the time of the event, that field did not have that value.

But it's all semantics really at that point.

Anyways, a relational database is a poor solution for this type of data. The stored data gains little to nothing, and may even negatively affect it's integrity (at time t, the event DID have this ID; it DID have this label), when stored relationally. Each event is discrete and there will be many of them which optimizes better for scale than relational organization.

I guess my point was there is vastly more useful data appropriate for a non-relational database than there is for relational databases. You might say it still has a "relation" in an abstract sense but this data does not need relational semantics within the database it resides in.

Re:MongoDB gets support for multi-document ACID transactions by ZenoArrow on Thursday February 15, 2018 @04:25PM (#16387774)
> "I believe Datomic to be the game changing database"

Have you used it? If so, what was your use case?

Re:MongoDB gets support for multi-document ACID transactions by woah on Thursday February 15, 2018 @12:47PM (#16385954)
As a developer: What? Almost everything is relational. I do appreciate Mongo's query language and ease of use (it was the first DB I learned), but your statement is ludicrous. Think about a basic blog system. You'll have relations between authors, posts, categories, and comments.

In my experience, Mongo is most often used with ORMs that emulate joins, like Mongoose. And the possibility of data inconsistency due to lack of transactions is ignored, or patched over with cleanup scripts after the fact.

Re:MongoDB gets support for multi-document ACID transactions by mason55 on Thursday February 15, 2018 @11:19AM (#16385179)
> otherwise you wouldn't be capitalizing on the advantages of JSON

Shouldn't this be "otherwise you wouldn't be capitalizing on the advantages of a schemaless, document-oriented database"?

The fact that it's JSON (vs. some other format) seems immaterial

Re:MongoDB gets support for multi-document ACID transactions by matwood on Thursday February 15, 2018 @11:29AM (#16385270)
As an aside, I hate the term 'schemaless' because there is always a schema.

https://blog.jooq.org/2014/10/20/stop-claiming-that-youre-us...

Re:MongoDB gets support for multi-document ACID transactions by lukaseder on Friday February 16, 2018 @05:50AM (#16391458)
Yep. Schema-on-read (E.g. MongoDB) or schema-on-write (E.g. Oracle)
Re:MongoDB gets support for multi-document ACID transactions by metheus on Thursday February 15, 2018 @11:33AM (#16385316)
Quibble: “schemaless” isn’t accurate, MongoDB supports schema (JSON Schema, in fact). We often say “dynamic schema”.

But your main point is good—- JSON isn’t the essence of it.

And yet it’s part of it, in that you could store XML and still be a document DB; MongoDB definitely chose JSON as the medium deliberately.

Re:MongoDB gets support for multi-document ACID transactions by ZenoArrow on Thursday February 15, 2018 @04:24PM (#16387770)
How does MongoDB handle schema changes? For example, let's say I want to add a mobile phone field to a customer record type. How would I go about doing that in MongoDB?
Re:MongoDB gets support for multi-document ACID transactions by drmirror on Thursday February 15, 2018 @07:15PM (#16388940)
The short answer is: just do it. You can add any field to any document at any time. That's the beauty of JSON documents without schema constraints. Then of course you need to let your application understand that. But it turns out it's almost trivial to make an application display a phone number field if it finds one, and not display a phone number if there isn't one in the document.
Re:MongoDB gets support for multi-document ACID transactions by mustardo on Thursday February 15, 2018 @07:40PM (#16389043)
This is true

It hurts when the next requirement comes along something like...

"As a user I want to have a home, work, and mobile phone number"

Now you have 3 "versions" of your implicit "schema" to contend with

1) No phoneNumber 2) phoneNumber and mapping it into / out of one of the three phone numbers in the UI 3) objects with three properties homePhoneNumber, workPhoneNumber, mobilePhoneNumber etc

Then the business comes up with "As a user I want to have arbitrary phone numbers that I can label" now the developers start to squeal

RDBMS + SQL is no panacea but having DDL operations like the following (all probably syntactically invalid but you get the idea) out of the box is incredibly powerful. ALTER TABLE user RENAME COLUMN phone_number TO home_phone_number; ALTER TABLE user ADD COLUMN work_phone VARCHAR(32) NOT NULL; CREATE TABLE phone_number (id BIGINT NOT NULL, user_id BIGINT NOT NULL, name VARCHAR(64) NOT NULL, phone_number VARCHAR(32) NOT NULL);

I have had reasonable success using MongoDB as a store of "things that happened" and will never change

Re:MongoDB gets support for multi-document ACID transactions by drmirror on Thursday February 15, 2018 @08:13PM (#16389186)
And I would still claim that this is easier in MongoDB because several versions of the phone number field(s) can happily coexist in the same collection. Those variants are usually trivial to understand for someone who even just looks at the data, and the application can be written to either accept the different formats, or adjust the format on the fly when it encounters a document that still uses an old schema. Or you could indeed write a batch job that bumps all your phone numbers to a new format, and you could put a JSON schema constraint on your collection that enforces the new schema for every future document. All those possibilities exist, and I truly see that as a big advantage.
Re:MongoDB gets support for multi-document ACID transactions by mustardo on Thursday February 15, 2018 @06:34PM (#16388744)
The onerous is mostly on the developer...

For any "real" system that is going to be in production for a long time this becomes a real problem

There are tools to "migrate" data but they come with all the limitations of the Mongo isolation model

Typically you either

* Write ad hoc (possibly using some tooling) code to iterate over your old data adding or mutating the field(s) in question

* Write queries such that they can handle the data being present, absent or in different forms for all of time. As you could expect this is a large burden

Re:MongoDB gets support for multi-document ACID transactions by Waterluvian on Thursday February 15, 2018 @10:21AM (#16384684)
That's what I was wondering too. It feels like SQL and NoSQL will meet in the middle: JSON with SQL support, SQL with JSON support.
Re:MongoDB gets support for multi-document ACID transactions by egeozcan on Thursday February 15, 2018 @10:26AM (#16384747)
"Mongres is a PostgreSQL extension that runs a custom background worker speaks mongo wire protocol."

https://github.com/umitanuki/mongres

Re:MongoDB gets support for multi-document ACID transactions by btown on Thursday February 15, 2018 @10:57AM (#16384988)
There's also https://www.torodb.com/stampede/docs/1.0.0-beta3/relational-... which tails the Mongo oplog and makes a fully-relational read replica, adding columns and indices as needed. An amazing (and free) shortcut to using analytics tools if Mongo's your main datasource.
Re:MongoDB gets support for multi-document ACID transactions by mcintyre1994 on Thursday February 15, 2018 @10:26AM (#16384743)
This is basically what Microsoft's Cosmos DB is, you store JSON documents, don't have to define a schema and you can query them with SQL.
Re:MongoDB gets support for multi-document ACID transactions by malloryerik on Thursday February 15, 2018 @11:45AM (#16385436)
And you can also bake in graph relations, right? How has your experience been with CosmosDB?
Re:MongoDB gets support for multi-document ACID transactions by mcintyre1994 on Thursday February 15, 2018 @12:14PM (#16385688)
To be honest I've only used it for pretty trivial things, I didn't do any joins. From the docs [0] it looks like they only have self joins, so joining on children or something like that but not across documents/collections.

Depending how much graph relation stuff you need you might be better off just using the graph API. I have no experience with that though. Or they support a MongoDB API if that covers your needs too.

Like I said I've only done basic stuff, but I really liked it - it's performant and really easy to set up and use. I used the Python API and it was really easy, then I switched to the Node one to try using it in Azure functions (Python library imports aren't really supported there) and that's nice too - it uses promises and works great. It also doesn't feel like a giant lockin (IMO) - their APIs work anywhere and there's no magic in Azure AFAIK to make you put your compute there if you're using the Database.

[0] https://docs.microsoft.com/en-us/azure/cosmos-db/sql-api-sql...

Re:MongoDB gets support for multi-document ACID transactions by overcast on Thursday February 15, 2018 @10:25AM (#16384726)
Which is why I'm depressed RethinkDB failed to take off :(
Re:MongoDB gets support for multi-document ACID transactions by sixdimensional on Thursday February 15, 2018 @11:37AM (#16385347)
Just remember, SQL is and always was "not only relational" - a play on the, should be long dead and buried acronym NoSQL (IMHO). Structured query language - I know for a fact from actually doing it (see Cache database, Hadoop schemaless SQL, even a product I worked on that was SQL to Mongo, heck the SQL standard itself) - guess what - it works with object/document stores too!

Mongo could have implemented SQL on top of their storage engine a long time ago minus the joins. Instead they built their own query mechanisms. Mind you, mapreduce can't be reproduced explicitly in SQL, but SQL expressions can compile to Mapreduce (see Apache Hive), so even that was not an excuse.

Edit: NoSQL served a purpose to remind people that there were other options other than relational databases (including those that predate the relational model and those that came after it), but man, what a terrible and misleading misnomer.

Re:MongoDB gets support for multi-document ACID transactions by munk-a on Thursday February 15, 2018 @02:04PM (#16386569)
On this point, the datastores that have insisted on using a query language that isn't SQL have always seemed to just be attempts at setting up walled gardens. Plenty of people have extended SQL when their needs required it (MySQL, PostgreSQL, Oracle, TSQL/MSSQL) but intentionally rejecting the basic format of SQL queries just seems like an attempt to setup a barrier to ever move off your DBMS.

Speaking as someone who ported an application from MySQL to MSSQL there is still a lot of work required to remove those custom extensions, but the core of what you're doing can remain the same.

Re:MongoDB gets support for multi-document ACID transactions by scarface74 on Thursday February 15, 2018 @11:26AM (#16385241)
I really don't need Sql compliance - I use C# and the Linq provider is pretty good.