If you have semi normalized data relationships, Mongo really doesn't fit the bill that well as your primary data store.
Where it shines is when you've got computed pieces of data to display (parts of a user data feed for example, data for a live graph, etc), you want to quickly store some data to be processed later (analytics, event processing), or you just use it as an object cache for expensive queries.
I have found that with a decently tuned DB, and I mean barely tuned, a decent data model, and sane fetching strategies (don't eager load everything always), a database can be good enough for quite a while, if not forever.
The answer I pretty much always get from people when I ask, "Why did you start with (whatever NoSQL they chose)?" Is, "Because (whoever) said it's faster and scales better."
I think younger devs that have options besides relational DBs find the barriers to entry on NoSQL lower (it looks mostly like how it went in and it's just JSON) so they pass over the learning SQL and basic data normalization fundamentals.
Not having to think rigorously about the logical structure of your data is a huge selling point for many. For programmers, because it means they don't have to think. And for employers, because it means they can make do with programmers who don't want to think.
I'll play the devils advocate: what ever logical structure of the data you come up with will not line-up with relational DB tables - I have had the pleasure of designing relational schemas for cyclic graphs. Hell, we have been trying and failing to do Object-Relational Mapping for years now, and it's still a shit-show. I could name a long list of ORM libraries and an even longer list of their various deficiencies.
Say you've thought rigorously about the logical structure of your data and v1 is out, and now a change comes along and you need an extra field or 5. Renormalize your structure, write some migration scripts (in both directions, just in case you want to downgrade). With NoSQL I'll just dump the new document structure and call it a day (or rather, deal wiht the differences in the application layer).
> I'll play the devils advocate: what ever logical structure of the data you come up with will not line-up with relational DB tables
That contradicts my experience. The relational model isn't a panacea, but it lets me model most scenarios adequately. Whenever the relational model is deficient, what I want is more powerful schemas (sum and product tables, constraints spanning several tables, etc.), not less.
> Hell, we have been trying and failing to do Object-Relational Mapping for years now, and it's still a shit-show.
I'm inclined to think the problem is actually at the object-oriented end of the bridge: https://news.ycombinator.com/item?id=11861520
> With NoSQL I'll just dump the new document structure and call it a day (or rather, deal with the differences in the application layer).
Correctly managing these differences without help from schema-aware tools is rather painful.
Sorry, but I don't want to work in an environment where programmers don't want to think or don't have to think. That is a scary thought. Has nothing to do with being locked into the past or being closed-minded. Just the opposite actually.
In other words, the application probably needs to deal with normalization and schema if it's taking copies of data, and using a relational database doesn't remove this requirement. (Of course, small applications often get away without enforcing schema, with the understanding that frequent database write failures due to malformed data are ok.)
Pretty much the classic http://www.mongodb-is-web-scale.com/?
My point is that we deal with these problems out of necessity. If you don't get a ton of traffic, then sticking with a normalized RDBMS is probably best. However, large scale systems will have redundant logic. Generic solutions (letting the normalized DB do the work) will often be slower than specific ones (denormalizing your schema and moving logic into the application layer). These are pretty simple principles that I've seen on almost every large-scale application I've worked on.
They are denormalized data, sort of.
"An index is a copy of select columns of data from a table that can be searched very efficiently that also includes a low-level disk block address or direct link to the complete row of data it was copied from." [1]
When you reference a field in an index, you're effectively duplicating this column on disk.
We can discuss semantics, but at the end of the day, indexes are glorified copies of your columns in a specific order.
- It's done at the hard disk level, so no database or data schema is involved.
- It's not done for performance reasons, but for security reasons (not to lose any data).
"In computing, denormalization is the process of trying to improve the read performance of a database, at the expense of losing some write performance, by adding redundant copies of data or by grouping data."
It's a technique that involves copying data for some very particular reason, an abstraction to hide the nasty duplication because the desirable thing is to have only one source of truth.
So while there's some remote similarity (the copying of data), the goals are completely different.
Uhm, no. Data warehouse is not a database with operational (i.e. current) data, it's a database with historical data. If it is out of sync with production for a day or a week, it's totally OK, depending on its design.
So no, cache invalidation is not a solved problem there.
You're assuming that you need to change things. Almost all big-data approaches work better when you abandon that and go for a log-structured model where you only ever append.
> I think the real issue is that "if you have a hammer, everything looks like a nail." Mongo and other NoSQL stores have some real use cases, but people who are more familiar with Mongo than RDBMS are too trigger happy to employ it as a solution to problems where a RDBMS is the clear solution.
True enough, but I think the converse is more true. If you're using a traditional RDBMS you're accepting a big series of constraints in exchange for functionality you often don't use. Indices are updated synchronously on every insert, slowing your writes, for the sake of transactional guarantees that most applications aren't written to take advantage of. Queries have to be passed over the wire as strings, so your application will spend a significant chunk of its time building them (hopefully using a library without vulnerabilities to send them over the wire so the database can spend a significant chunk of its time parsing them), or else you get to deal with the db-specific and per-connection quirks of prepared statements, for the sake of supporting ad-hoc querying in a language that frankly isn't great for humans. Tables, materialized views and the like occupy this awkward in-between condition where it's not clear whether you're supposed to create and modify them ad-hoc (and manipulate them programatically) or not; if you really do need to do ad-hoc reporting then they're what you want, but often the performance implications of that make it unacceptable to do in production.
I don't think a lot of cases are good fits for RDBMS. If you don't need ad-hoc reporting then you're better off building a data processing pipeline where you produce your results directly rather than the sort of semi-aggregating you end up with with an RDBMS. If you don't need full ACID then it's not worth paying the performance cost of it. And I don't know why there aren't RDBMSes without better query languages and schema definition languages, but there aren't.
Or you could use stored procedures. Think of it as making your RDBMS into a microservice, if you like.
>>I don't know why there aren't RDBMSes without better query languages and schema definition languages Huh... I always thought SQL was pretty straightforward for most cases. It only gets really arcane when you get into advanced cases (eg, recursive queries) and/or vendor extensions. It does take a bit of a mental shift, but no more so than imperative-style to functional-style, IMO.
There are certainly tradeoffs to using an RDBMS. But I think a lot of people jump to the conclusion that they have a big-data situation when they really don't. I also think a lot of people underestimate the value of very easy ad-hoc querying and already-solved backup/restore.
The trouble is testability, versioning, deployment, and SQL just not being a pleasant language for expressing business logic. And the lack of a library ecosystem. And poor IDE support. And...
No kidding - don't do that. Sprocs are great to present a query interface and avoid sending queries over the wire every time. They can be handy to decouple the actual storage architecture from the query interface, so you can do tricky stuff in SQL without screwing up the clients. That doesn't mean you should stick your business logic in there.
>>testability, versioning, deployment... library ecosystem...
And how would any of those problems be solved by using raw SQL strings or a document DB? That just moves the problems back into the schema or the data, where it's even harder to deal with.
>>poor IDE support
So your code editor has a better IDE for SQL? Or are we talking about a document DB? I haven't seen any of them that were anywhere close to SSMS.
I'm arguing for using a non-SQL interface. Either a structured binary query protocol (which leaves you shipping a lot of data around sure, but at least removes the constructing-and-parsing SQL overhead), or a map-reduce style setup where you can run your queries where the data is but in a first-class programming language.
How is a binary query protocol fundamentally different than calling a sproc?
How is a map-reduce job in Javascript any different on the attributes you mentioned than ad-hoc SQL or a sproc?
Obviously map-reduce is a fundamentally different approach to data processing, but it seems to have similar traits in regards to source control, testability, etc.
Switching to a model where you supply your own map-reduce or pipeline is a real shift, I think, from the DB as a framework that manages querying for you to more of a library/toolkit you can use to write your own computations. The indexed tables model is an incredibly effective compromise, but it's still a compromise - if you know specifically what you need to do, you can do it better.
Running a first-class programming language with full support for usual programming language tools is a major difference for developability, testability, libraries and so on. Deployment model depends on the datastore - plenty of non-SQL ones have room for improvement here - but I think the traditional RDBMS still has the worst of it.
Most of my experience has been with SQL Server, which I think does use a (mostly) binary wire protocol, at least when calling sprocs. It also has a pretty good query optimizer - I've spent a lot of time trying to beat it with hand-tuned queries and only come up with something better about 50% of the time. And there are a lot of tools for source controlling the DB scripting and testing everything. It's not hard to do unit test style runs that set up and tear down tables and DBs.
Has that not been your experience?
Hmm. I'm used to invoking storedprocedures via "select mysproc(param1, param2, ...)" which still has to be formed into a string and then parsed on the DB side using the arbitrary-SQL parser (because there's no way for the DB to know a priori that it's not a "regular" query). Does SQL server have some special case binary protocol for invoking them directly?
> It also has a pretty good query optimizer - I've spent a lot of time trying to beat it with hand-tuned queries and only come up with something better about 50% of the time.
The query planner can usually run the best query possible with the indices that exist, sure. But you have to fit your calculation into this model of indices that are built on insert and everything else happening at query time. Or you go down the route of lazy materialized views that make use of other lazy materialized views, specifying indexing strategies... I mean I think you can ultimately express any data processing pipeline in a RDBMS if you try hard enough (though you have to use database-specific features that tend to be less-well supported by the ecosystem), but at some point it's easier to just have a first-class programming language that has access to the data, and write the code that you want to run.
> And there are a lot of tools for source controlling the DB scripting and testing everything. It's not hard to do unit test style runs that set up and tear down tables and DBs.
Up to a point. It's easy to end up with "unit" tests that take a second for each test, which mean it's not really practical to get good coverage of logic there.
In terms of source controlling and so on I guess the big problem is that you now have a distributed system written in two quite different technologies. So you've got to figure out a release and deployment process that handles both, and a lot of shops don't seem to bother. If you're already running a multi-language microservice architecture then this is probably a lot less of an issue.
>>So you've got to figure out a release and deployment process that handles both, and a lot of shops don't seem to bother.
Yeah, that's true. Setting up something along the lines of Rails' migrations isn't really that hard, but many don't bother.
> The trouble is testability, versioning, deployment, and SQL just not being a pleasant language for expressing business logic.
Most of those, to the extent they are valid, are trouble with using stored procs in SQL for that purpose, but most RDBMSs support stored procs in languages other than SQL.
Those are implementation details, not inherent properties of the relational model.
Well, better is subjective, but there are RDBMSs with languages other than SQL (e.g., Dataphor with its D4 language.)
That depends on your loads. If you're write heavy, then yes. If you're read heavy, then you're adding extra work to find the current row out of all the appends. If you're read and write heavy, then get used to buying really expensive beefy machines and maybe sharding...
I think a good solution is to use something like DynamoDB and use hot/cold tables. Like, if you're storing something high read/write (i.e. today's olympic stats), keep it in the hot table with a frequent cache timeout. Later, when they become yesterday's stats, move them to the cold table which will basically be read-only, with a longer valid cache.
The problem with MongoDB is that it discourages this type of data management, at least how I've seen it used.
CouchDB does this to a certain extent, but its JSON-document approach is not appropriate enough.
Regarding it as essentially a form of caching has been helpful. In fact there is a spectrum that goes from simple caching to denormalized schema with many points inbetween.
NO VIEWS! NO STORED PROCEDURES! NO TRIGGERS! YOU SQL MUST BE AGNOSTIC!
and stuff like that. However, they pick Mongo or anything like that and yep, they must use anything it provide just by necessity.
---
People:
Using views is easy. Is probably the most simple thing you can get to leverage what a DB can give (plus indexes) and let model the data in different ways and even help in transition from a data model and another.
That's a trade-off you have to analyze to see if it is worth it. If that data is not maintained often then is OK to have duplicated data. One thing you have to realize is that there is no a silver bullet in the CAP theorem and you always would give up something.
The other thing to realize is that not everybody needs a distributed database and most problems will live happily with an old-fashioned relational database.
I don't like MongoDB either, but normalization is not the answer for every project. For some projects, reading efficiency is so important that people are willing to pay for of it in disk space, an increase in writing costs, and even app complexity.
I think the bigger issue is that people are afraid of separating data at clear boundaries, so long as the orchestration is in place. Having definitive data in SQL, and having denormalized documents in elasticsearch/mongo/rethinkdb works very well... other instances (huge read and/or write throughput) may work better in a Cassandra (or other bigtable solution) with application-generated lookup tables may be better. It just depends on your need.
With better support for JSON/XML nested structures and indexes in sql rdbms, it's becoming less of an issue for a lot of types of data/documents.
Like I said, it was pretty crushing on the database... there were a lot of other expensive calculations that ran per request... pre-calculating, caching, and a few other things got some longevity, bit it was a tangled mess of a beast for what would have been 3-4 collections for a document database (mongodb, rethinkdb, etc).
NoSQL is designed to support both of these scenarios well. And with lower overhead requirements by not having to update a central store... It is used in particular for Big Data solutions. Rather than having to be optimized for one or the other as an RDBMS would require.
Normalization also better supports ad hoc inquiries and reporting compared to denormalization; denormalization is an optimization for regularized reporting patterns (responding to a question that a denormalized reporting design didn't anticipate often requires effectively renormalizing the data, which recovers the logical benefits of normalization, at a potentially very high performance cost.)
> NoSQL is designed to support both of these scenarios well.
Without the integrity benefits of normalization on updates, and without the benefits of normalization for ad hoc queries.
OTOH, there are very few uses of normalized RDBMSs where only the write performance benefits are sought -- usually at least integrity is a concern, and sometimes ad hoc querying as well.
Not just duplication. I did some work on a platform that used ElasticSearch as a relatively relational "document" store (that is, a purpose to which a true RDBMS would have been better suited). Besides data duplication, we ran into the problem of data consistency: using the author's TV show example, we would end up with some actor's whose birthdates would be recorded, but some whose age was recorded instead. Some TV shows would list the actors by role, some would just list the actors and the roles in two different lists. Sometimes the TV show start and end dates would be milliseconds since the epoch, some would be dates in YYYY/MM/DD formatted strings. Some would be dates in DD-MM-YY formatted strings. All scattered randomly throughout the data store. Elastic schema means random schema.
Of course we (they) were - lots of it! Just inconsistently...
I've been using Mongo recently, and every time I raise criticism of it, the counterargument I hear is "forget about normalization! Duplication is okay."
Like the author, I really cannot wrap my head around this. While I understand that duplicating documents across collections may make querying faster, what about when you want to change the document? You need to propagate the change across every duplicate of the document in every collection where it exists. This means that any "de-duplication" logic needs to happen at the application level, rather than the database level.
Parse.com provided an interesting abstraction around that with "cloud code" and beforeSave and afterSave triggers. To maintain consistency, they encourage propagating updates to a document within its collection's beforeSave function. So if a document changes in one collection, you write code to change all instances of the document in other collections. That's nice in that it almost feels like you're writing the dedeuplication logic in the database layer, because you can view the "beforeSave" and "afterSave" functions as extensions of the schema. As long as those functions are up to date, the schema and any pseudo-linked documents will stay up to date.
But I really don't buy it. The strengths of mongo encourage a design that necessitates complexity for any significant write operation.
I think the real issue is that "if you have a hammer, everything looks like a nail." Mongo and other NoSQL stores have some real use cases, but people who are more familiar with Mongo than RDBMS are too trigger happy to employ it as a solution to problems where a RDBMS is the clear solution.