News for hackers. Stuff that matters.
Re:The Marketing Behind MongoDB by martinald on Tuesday August 29, 2017 @01:30PM (#15125984)
100% of my friends who have used Mongo/similar NoSQL have given up and had a nasty rewrite back to pgSQL.

This seems to be the journey:

1. Lack of migrations is awesome! We can iterate so quickly for MVP

2. Get users

3. Add features, still enjoying the speed of iteration

4. Get more users

5. Start building reporting features for enterprise/customer support/product metrics (ie: when the real potential success starts)

6. Realise you desperately need joins, transactions and other SQL features

7. Pause product dev for 1-3+ months to migrate back to SQL, or do some weird parallel development process to move it piecemeal back.

I think the most interesting question though is would they be able to get MVP and initial customers that set off this if they were moving (slightly) slower due to SQL and slight overhead that comes with?

My thought is definitely yes.

Re:The Marketing Behind MongoDB by brandur on Tuesday August 29, 2017 @02:04PM (#15126282)
> I think the most interesting question though is would they be able to get MVP and initial customers that set off this if they were moving (slightly) slower due to SQL and slight overhead that comes with?

I've used Postgres and Mongo pretty extensively, and for any reasonably seasoned developer, the startup overhead of an SQL system is a myth. There may upfront cost to learning how an RDMS and SQL work in the first place, but once you're familiar with them, they'll be faster than Mongo on any new project.

The schemaless concept of a document database seems to be the major selling factor in velocity of movement, but once you've got a good handle on a migration framework in the vein of ActiveRecord or other popular software, that's negated completely. It also really doesn't take long before schemaless starts to cause big problems for you in terms of data consistency -- it's not just the big players that get bitten by this.

The simplified query language is another one. SQL is a little bit obtuse, but it's not that bad once you have a handle on it, and a lot of people are familiar with it. Once you add in an ORM layer, the lazy-style access of a framework like Sequel or SQLAlchemy makes the developer experience quite a bit better than any Mongo APIs that I've seen. Also, after you get beyond trivial usage, SQL's flexibility so wildly outstrips Mongo's query documents that it's not even worth talking about.

Postgres on the other hand ships with a great management CLI, a very powerful REPL (psql), and features like data types/constraints/transactions that guarantee you correctness with zero effort on your part. I can only speak for myself, but I'd take Postgres to the hackathon any day of the week.

Re:The Marketing Behind MongoDB by martinald on Tuesday August 29, 2017 @02:37PM (#15126573)
I totally agree with you, and started writing something about how understanding a good ORM takes nearly all the headache away.

I think the thing people do find slow is a lot of 'documents within documents' in SQL. It turns out this is usually a bad development pattern long term but it is super fast being able to just add docs inside docs with no configuration. It feels very slow writing foreign keys, navigation props and schemas for this in SQL vs JSON, where you can just dump your object in and you're done.

Basically; I think with noSQL you get some very short term gain for a lot of long term pain, and you're right, ORMs and other tooling solves this mostly.

I myself fell for this trap, and while it was a nightmare it actually matured me more as a professional more than anything I've ever done recently. Regardless of crazy hype, I don't think I'll ever fall for a solution so easily without evaluating it properly.

I think I assumed the "crowd" had done the tech due diligence on this stuff and it definitely wasn't the case.

Re:The Marketing Behind MongoDB by macintux on Tuesday August 29, 2017 @03:46PM (#15127058)
I agree that Postgres trumps Mongo for most use cases, but if ORM is the answer, you might be asking the wrong question.

I love that Fowler discusses[1] how to avoid ORMs and offers only two possibilities:

> Either you use the relational model in memory, or you don't use it in the database.

He clearly has a blind spot. The correct answer, at least in some cases, is don't use objects in the first place.

[1]: https://www.martinfowler.com/bliki/OrmHate.html

Re:The Marketing Behind MongoDB by anentropic on Tuesday August 29, 2017 @05:18PM (#15127681)
I've been quite skeptical of the anti-ORM sentiments, since I always found the ORM handy, not a problem etc.

Two things recently started to change my mind

One is, after profiling some views in our Django app I found that a significant portion of the time in list views (i.e. returning many objects) was spent instantiating ORM model objects. Not on fetching them from the db, but turning them into objects whose imminent destiny is to get turned back into dicts for serializing to JSON.

So we switched to values() queries which return plain dicts, but then the ORM is just a query engine.

The second was, in a large app, finding that many of the tables start to require a degree of care to only perform 'safe' queries, i.e. ones supported by appropriate indexes. The integration of the ORM into the framework provides many conveniences, but this tends to involve a lack of awareness/control over exactly which queries get made. Most of the patterns that are 'idiomatic' to the framework also tend to involve a lot of ORM querying logic pushed down in the view layer... while our need to perform only a restricted set of 'safe' queries pushed us in the other direction, towards a repository layer with carefully curated query code.

I'm seriously considering exploring ORM-less development in future

Re:The Marketing Behind MongoDB by wvenable on Tuesday August 29, 2017 @05:31PM (#15127762)
There's no single technology that's perfect for all situations. An ORM is really useful for a lot of very common tasks. Most apps include a lot of reading/writing of single rows and basic list queries for which ORMs are a superior way of working.

But for anything involving complex querying, I typically just go straight to SQL. Most ORMs provide plenty of ways to handling different levels of query need from full ORM queries, to queries into entity objects, to queries into flat dictionaries.

I don't think ORM-less development is the answer to the wisdom that an ORM isn't a complete and sufficient interface to the database. That's just throwing the baby out with the bathwater.

Re:The Marketing Behind MongoDB by sjellis on Wednesday August 30, 2017 @08:32AM (#15131538)
"The integration of the ORM into the framework provides many conveniences, but this tends to involve a lack of awareness/control over exactly which queries get made."

I think that the lack of visibility is the problem, more than ORMs as such - one of the things that Rails got right is the logging.

In development, Rails will show you every SQL query as it happens, so you can see the blizzard of SQL queries that your request-response cycles are generating, and can spot crazy stuff once you have conditioned yourself to actually read the logs. Conversely, I've found that not being able to easily see what the ORM for your platform is doing can be really disempowering.

I suspect that nicer graphical tools would make this much more accessible: a lot of developers seem to either treat logs as noise, and I've seen some junior devs appear almost afraid of them.

Re:The Marketing Behind MongoDB by anentropic on Wednesday August 30, 2017 @10:40AM (#15132461)
It's easy to get logs, it'd be better not to deploy code with unsafe queries and then have to scan through the logs to see why your database is on its knees though
Re:The Marketing Behind MongoDB by Can_Not on Wednesday August 30, 2017 @08:49AM (#15131628)
I haven't thought of this before, ORMs probably use 2-3x as much memory as regular structs/dicts. I've started using only the query builder in all my side projects and I've had a good experience so far.
Re:The Marketing Behind MongoDB by anentropic on Wednesday August 30, 2017 @10:45AM (#15132500)
yes, not just the memory... for models with lots of fields, in Python+Django, the time overhead of instantiating these complex heavyweight objects can really add up if you have a whole 'page' of them

I always figured the main overhead was doing I/O with the db, but in these cases it's not always so

interesting blog post about this from the creator of SQLAlchemy: http://techspot.zzzeek.org/2015/02/15/asynchronous-python-an...

Re:The Marketing Behind MongoDB by chipotle_coyote on Tuesday August 29, 2017 @07:14PM (#15128414)
The correct answer, at least in some cases, is don't use objects in the first place.

That at least fits my interpretation of what Fowler meant by "use the relational model in memory," though--if you're using a relational database, don't try to directly map objects to relational tables, but take advantage of what SQL offers instead. That's consistent with not using objects in the first place (although I suppose it doesn't require it).

Re:The Marketing Behind MongoDB by macintux on Tuesday August 29, 2017 @07:23PM (#15128473)
I may indeed have interpreted that uncharitably.
Re:The Marketing Behind MongoDB by EGreg on Tuesday August 29, 2017 @07:48PM (#15128595)
Why do people always trot out the SQL relational databases?

If I had to do it all over again, I'd use a graph database. They work better and are faster for most web apps and social stuff. Including reports and joins.

And in my experience, I have avoided joins and used the database mostly as a key value store. Which makes it perfect for when we need to scale. We just make a CockroachDB adapter and that's it!

Re:The Marketing Behind MongoDB by mohaine on Wednesday August 30, 2017 @09:18AM (#15131869)
The issue is that if you can avoid joins long term, then you really either don't have enough data, enough complexity, or very rarely managed to have a data set that doesn't need joins.

If it is option 1 or 2, you just didn't need joins YET. When the time comes that 1 and 2 no longer apply it will be VERY hard to add them.

Sure you can use both types of databases where it makes the most sense but at that point you might as well just use the "NoSql" functionality in the relational db. PSQL does a JSON doc store VERY well and even allows deep joins when time comes.

I personally never saw much benefit to the graph databases. Sure the are great for a few data structures but most of the speed differences came them from being in memory more than anything else. Pretty much everything is fast with a good data structure design and everything in memory.

Re:The Marketing Behind MongoDB by thehardsphere on Wednesday August 30, 2017 @11:27AM (#15132932)
I have noob questions about what you've written here. I apologize for asking them here but you seem like you think clearly and know what your are talking about.

If you are in case 3, then is that when NoSQL allegedly starts paying off? If so, is the payoff in actual performance or just not having to write SQL?

I know you said that you could just use the NoSQL features of the relational DB. If you really truely know for certain that you will never need to join your data, then isn't using an entire relational DB software package a lot more overhead?

Re:The Marketing Behind MongoDB by mohaine on Wednesday August 30, 2017 @01:45PM (#15134267)
I case 3 you get get some real speedups, but you usually have to design the db layout for the use case. Step one is "How am I going to lookup this data" then design the data structures for that use case. Often(but not always) in these cases the NoSql db is just a lookup db with the system of record being a somewhere else and data is transformed into multiple data sets for lookup.

IMHO, most of the NoSQL hype was developers not wanting to learn sql/ddl along with a lot of resume padding. It is also important to note that NoSQL no longer means "No SQL" but really is now "Not Relational and/or ACID" (https://en.wikipedia.org/wiki/ACID). The only thing worse than SQL is everything else created to replace it. Many "NoSQL" products use a limited SQL like syntax as their only query format.

As for relational DB overhead, there is no reason for it be more then NoSQL(see sqlite) unless you are willing to give up part of ACID. Special cases like this are another use case for NoSQL but you probably want ACID unless you have data/throughput to make it nonviable. Otherwise you have it just ignore issues (normally what happens) or code for them in your app.

Re:The Marketing Behind MongoDB by EGreg on Wednesday August 30, 2017 @10:58AM (#15132642)
Actually major graph databases store data on disk as well. That's not the reason they are faster. The reason is that all the pointers point directly to the data.

Consider how you'd get the "top 10 books, and their related authors, and their related biographies".

1. A graph database would literally just look at an index of books, and then grab the book records. A relational database would do the same with an index, so far so good.

2. Now comes the difference. For each book, the graph database would just load the list of pointers to related authors and load those. And for each author, it would load the list of pointers to related biographies, which have pointers to related pictures etc. And in O(jk) where j is the number of books to return, and k the maximum number of things to get per book, it's done.

3. Now consider the same step for a relational database. After getting the books, it has to load the authors, and search it for each author. This takes O(k log N) where N is the total number of entries, and grows (albeit slower and slower) with increasing amounts of data. Then once every author record is loaded, it has to enumerate all their ids into a giant list and do it again. The list can be stored incrementally and the searches can be parallelized but at the end of the day ALL JOINS have an extra O(log N) factor, which is what slows down the database, usually by a factor of 10 for data that's in the millions or billions of rows.

4) The design of a graph database naturally encourages using indexes. In relational databases you have to remember to add them. And even after you do, the relational database takes log N longer to do all the joins. And joins are done very often in social networks, fetching related stuff, and other things in a normalized schema.

You can do all the relational algebra stuff while walking a graph, too.

In fact any relational database can be turned into a graph database by just adding a variable-length list of "pointers to related data" to each record which would point to the actual location of the rows in the related table's index. And then manage all relations between rows not as joins but as entries in these lists. And finally, implement support for a graph database language alongside SQL. However, I have not seen any such extensions to InnoDB or Postgres etc. which turn them into graph databases.

Re:The Marketing Behind MongoDB by mohaine on Wednesday August 30, 2017 @01:55PM (#15134360)
I understand that they do persist to disk and store data differently but the ones I've seen used in practice are in memory during runtime. It as been a few years but so maybe this has changed. Any examples?

Graph databases also are only fast when doing graph transversal lookups. They all have index just like RDB. Sure you got from parent->child->grandchild quickly but how to do you find the parent in the first place? An index normally.

Re:The Marketing Behind MongoDB by EGreg on Wednesday August 30, 2017 @06:28PM (#15136648)
But the parent is one lookup and the children are 1000. Thats why.
Re:The Marketing Behind MongoDB by mohaine on Friday September 01, 2017 @10:40AM (#15148671)
This is only if the 1000 children are already in memory. Otherwise it is still 1000 lookups, by index. This is why the Graph Dbs on the market tend to be in memory.

I'm not saying Graph DBs don't have their place. They do, but they are the wrong answer for MOST datasets.

Re:The Marketing Behind MongoDB by EGreg on Sunday September 03, 2017 @12:25PM (#15162254)
NO. Each lookup thereafter is NOT by index. That's where the savings come from. The pointers already contain the exact block where to load the info from.
Re:The Marketing Behind MongoDB by Can_Not on Wednesday August 30, 2017 @09:05AM (#15131748)
What's a good resource in getting introduced to "the postgre of graph databases", including it's value proposition over regular relational databases?
Re:The Marketing Behind MongoDB by EGreg on Wednesday August 30, 2017 @10:48AM (#15132530)
Re:The Marketing Behind MongoDB by SkyMarshal on Wednesday August 30, 2017 @04:02PM (#15135480)
From what I've observed, the "crowd" is easily seduced by performance above all other concerns - correctness, security, science, etc. NoSQL was invented and gained popularity b/c it originally was easier to scale via sharding, and that was seductive enough [1] to give it industry momentum. While performance is a feature, it seemed many folks originally advocating NoSQL did not understand or appreciate the mathematical foundations of the relational model. A decade of experience seems to be driving zeitgeist back to scientific fundamentals.

[1]:https://www.youtube.com/watch?v=b2F-DItXtZs

Re:The Marketing Behind MongoDB by lewisl9029 on Tuesday August 29, 2017 @04:07PM (#15127207)
> I've used Postgres and Mongo pretty extensively, and for any reasonably seasoned developer, the startup overhead of an SQL system is a myth. There may upfront cost to learning how an RDMS and SQL work in the first place, but once you're familiar with them, they'll be faster than Mongo on any new project.

> The schemaless concept of a document database seems to be the major selling factor in velocity of movement, but once you've got a good handle on a migration framework in the vein of ActiveRecord or other popular software, that's negated completely. It also really doesn't take long before schemaless starts to cause big problems for you in terms of data consistency -- it's not just the big players that get bitten by this.

I don't disagree with the overall sentiment, but this post is overstating things in a few places...

Schemas add friction in development. You can certainly minimize this friction with experience and tooling, but for a completely new project, no amount of experience or ORM magic is going to "completely negate" that additional friction vs a schemaless system where you can change the shapes of your data and the relationships between them without any restrictions whatsoever.

For rapid prototyping of new projects, it could be entirely reasonable to trade off long term data-consistency benefits of having a well-defined schema in return for faster iteration times. But taking that schemaless prototype to production directly is decidedly unwise, so the final step in a prototyping process like this should involve a refactoring of your data model to introduce a schema/migrations system before that point.

It would be fine to argue that that final refactoring step to make the schemaless prototype production-ready could completely negate the faster initial iteration times afforded by starting out with a schemaless DB (I too would generally agree with this assertion, but we must also remember not all prototypes will go to production, as prototypes are meant to prove out an idea, and the idea itself won't always work out). But to claim on a brand new project (where the data model is ill defined and prone to sweeping changes), one can iterate _faster_ while maintaining a schema and migrations system than on a schemaless system (given similar levels of proficiency with both systems) strikes me as hyperbolic.

Re:The Marketing Behind MongoDB by jdmichal on Tuesday August 29, 2017 @04:24PM (#15127320)
> For rapid prototyping of new projects, it could be entirely reasonable to trade off long term data-consistency benefits of having a well-defined schema in return for faster iteration times. But taking that schemaless prototype to production directly is decidedly unwise, so the final step in a prototyping process like this should involve a refactoring of your data model to introduce a schema/migrations system before that point.

I don't get it. If you're not in production anyway, you're not working with real data. And if you're not working with real data, you don't have to migrate anything. The entire problem just straight-up disappears. Just blow away your database and recreate it with the new schema. Modify your test data insertion scripts to match and load them.

Re:The Marketing Behind MongoDB by tomc1985 on Tuesday August 29, 2017 @05:24PM (#15127725)
Or alter your schema in the client. Quite easy to right-click and 'Add New Column' or ADD COLUMN or whatnot. I'm trying this with a new project and it is resulting in a data model that feels cleaner and hand-built.

One can simply capture the schema when they've found a good structure, and then use that as the starting point for a migration system (like ActiveRecord).

Re:The Marketing Behind MongoDB by lewisl9029 on Tuesday August 29, 2017 @04:37PM (#15127417)
The need to maintain an explicit schema and update it every time your data model changes is the point of friction here, not the migrations system. To be perfectly clear, I'm not suggesting maintaining a schema is onerous enough to outweigh the numerous other benefits having an explicit schema provides, even at the prototype stage, but when you're prototyping and don't need to worry about data migrations, having to maintain a schema is certainly not going to be _faster_ than not having to maintain a schema, as the original post claimed.
Re:The Marketing Behind MongoDB by wvenable on Tuesday August 29, 2017 @05:35PM (#15127789)
You always have to maintain an explicit schema -- it's just defined in your code instead of the database. And depending on your tooling, it's not easier/faster to maintain that schema in code vs. the database.

What is worse for schemaless databases is schema versioning. Your code will forever have to support every schema that has ever existed in production data. I have code to support schemaless data that was entered well over a decade ago. If I had simply used a relational database I would just have a single schema to deal with.

During development, schema changes are quite frictionless. It also provides extra safety and documentation that any friction is completely worth it.

Re:The Marketing Behind MongoDB by lewisl9029 on Tuesday August 29, 2017 @08:50PM (#15128866)
> You always have to maintain an explicit schema -- it's just defined in your code instead of the database.

This is normally known as an implicit schema. It's implicit precisely because you don't explicitly maintain a schema separately from your application code.

> During development, schema changes are quite frictionless. It also provides extra safety and documentation that any friction is completely worth it.

The second sentence contradicts the first. And I'm not disagreeing with the second. The friction may be worth it, but calling it _frictionless_ was my issue with this post and the original (which went even further by asserting the friction somehow accelerated development).

Re:The Marketing Behind MongoDB by wvenable on Tuesday August 29, 2017 @10:00PM (#15129176)
> It's implicit precisely because you don't explicitly maintain a schema separately from your application code.

But that's not completely true. There is plenty of code necessary for managing the schema and just the schema in a document database. Default values, relationships, validation, and managing previous version of the schema. I have plenty of code that just exists to handle documents in the "old" format.

> The second sentence contradicts the first.

Maybe I should have said almost frictionless instead of quite. And honestly, some things are less work with a schema (like creating a new defaulted boolean column) in an RDBMS than in a schemaless design. So the friction is relative.

> The friction may be worth it, but calling it _frictionless_ was my issue with this post and the original

I have a similar issue with calling a document-store _schemaless_. It's not schemaless, it has a schema. In fact, it has as many schemas as there are changes to the structure of the data. And this, in my opinion, is the biggest negative to that kind of design.

Re:The Marketing Behind MongoDB by lewisl9029 on Tuesday August 29, 2017 @10:38PM (#15129336)
I don't think we're really disagreeing on anything, just talking over each other a bit. Let me try to clarify myself:

> But that's not completely true. There is plenty of code necessary for managing the schema and just the schema in a document database. Default values, relationships, and managing previous version of the schema. I have plenty of code that just exists to handle documents in the "old" format.

This is only the case because you chose to _add_ an explicit schema on top of whatever schemaless DB you were using for maintainability reasons, which is certainly necessary if you're building a production app. However, in this case I was talking specifically about using schemaless DBs in the context of prototyping, and adding an explicit schema (that specifies things like default values, relationships, and migrations) is certainly not mandatory in the prototyping phase when you're not dealing with any real/past data or migrations.

MongoDB and other schemaless databases by default will happily accept whatever document you want to store in it without any care in the world, and this can be a desirable property for iterating as quickly as possible, but in production you definitely want to specify an explicit schema of some kind on top of Mongo as you have done, or just move off of Mongo altogether onto a proper relational database with a mandatory explicit schema.

As to which is the better long-term choice for a production app, I totally agree with everything you said. With schemaless databases, even if you add an explicit schema on top of it, there's always still the possibility to underspecify in your schema and re-introduce implicit data dependencies into your data model and application logic, which can lead to nightmarish bugs in production (I've experienced many instances of this first hand). It's much better to use a database designed around a mandatory explicit schema for relational data for the stronger data consistency guarantees they provide once you start handling real, persistent data.

Re:The Marketing Behind MongoDB by wvenable on Wednesday August 30, 2017 @12:39AM (#15129767)
When you're prototyping, you can be equally as sloppy with a relational database and make the same productivity gains. However, the sort of thing you describe doesn't even sound like a real prototype but rather something fairly trivial:

> by default will happily accept whatever document you want to store in it without any care in the world, and this can be a desirable property for iterating as quickly as possible

I fail to see how a database full of mismatched documents is valuable in prototyping. My own experience with prototyping in an RDBMS is just a constant evolution of the existing sample (sometimes real) data. Adding new columns is trivial, breaking up a single column to a one-to-many is a simple insert..select into a new table as one example. Similar transformation with a document-store involves writing a lot of throw away code.

But from another perspective entirely, I find actually designing the schema to be the best place start when I prototype an application. If I have the database design correct then designing the corresponding UI or API is almost trivial. Now obviously that's just one style of development but it's a no less valid one. And I don't have to do anything more to move to production.

Re:The Marketing Behind MongoDB by lewisl9029 on Wednesday August 30, 2017 @01:57AM (#15129982)
> I fail to see how a database full of mismatched documents is valuable in prototyping.

Again, as you yourself even brought up, just because you don't have an explicit schema doesn't mean your data needs to be schemaless. There can still be an implicit schema to your data that depends on the shape of the documents you store in your database. And in a prototype without real data (I'm defining real data here as data that can't be trivially discarded without consequence), the documents that actually end up in your database will have a uniform shape simply because your code only operates on the current version of that implicit schema, and older versions can simply be deleted.

> But from another perspective entirely, I find actually designing the schema to be the best place start when I prototype an application. If I have the database design correct then designing the corresponding UI or API is almost trivial. Now obviously that's just one style of development but it's a no less valid one. And I don't have to do anything more to move to production.

That approach is certainly valid. I just wanted to make it clear that using a schemaless database doesn't mean you can't do any schema design up front, it just affords you the ability to skip the overhead of updating and adhering to an explicit schema at every step in the evolution of your prototype. In order words, you let your code and your product needs drive the changes and growth in your schema, and only look towards solidifying the changes in your schema into an explicit document to adhere strictly to once you have more concrete insights into how your data model needs to look, driven by the data needs of a working prototype.

I feel I've stated my position as clearly as I could, so I'll leave it at that. Feel free to reply if you still take issue with anything I said, but I probably won't be responding.

Re:The Marketing Behind MongoDB by wvenable on Wednesday August 30, 2017 @11:17AM (#15132823)
I think you have stated it well. And I agree that there is overhead in creating a schema. I think, however, where we might disagree is on whether or not that overhead is anything more than trivial.

I think ultimately this is similar to the debate on static vs. dynamic typing.

Re:The Marketing Behind MongoDB by twic on Tuesday August 29, 2017 @04:52PM (#15127509)
If you're using an ORM, you can generate the schema automatically. With Hibernate, you set the hibernate.hbm2ddl.auto property [1]. If you set it to 'create' or 'update', it will alter the database schema so that it matches your objects, which is handy for prototyping, and terrible for production. If you set it to 'validate', it will check that the database schema matches your objects, and refuse to start if it doesn't, useless for prototyping but ideal for production.

[1] https://stackoverflow.com/questions/438146/hibernate-hbm2ddl...

Re:The Marketing Behind MongoDB by lewisl9029 on Tuesday August 29, 2017 @09:04PM (#15128933)
That's a neat feature, but as you may already know, ORMs come with their own baggage in the form of object-relational impedance mismatch. There's an entire rather-substantial Wikipedia article on it, so I won't go into it any further myself: https://en.wikipedia.org/wiki/Object-relational_impedance_mi...

The sibling thread also brings up similar issues with ORMs: https://news.ycombinator.com/item?id=15127058

Re:The Marketing Behind MongoDB by aryehof on Friday September 01, 2017 @09:32AM (#15148171)
ORMs are a solution to the problem of the impedance mismatch between an object model and its corresponding relational model. It's not that ORMs come with the problem.

I think it increasingly rare today to see real object domain models anyway - the original raison d'etre of ORMs. Instead it's common to see domain concepts modeled in the data layer and modified by logic (often limited to CRUD) in an application/service/controller layer. The problem with this of course is that as complexity increases, that app/service/controller layer increasingly approaches a big ball of mud.

Edit: grammar

Re:The Marketing Behind MongoDB by twic on Wednesday August 30, 2017 @04:44PM (#15135837)
Eh. I've used ORMs and SQL extensively for many years. I think the concerns over mismatch are routinely overstated.
Re:The Marketing Behind MongoDB by jimktrains2 on Tuesday August 29, 2017 @05:14PM (#15127650)
It's such a small amount of friction compared to the advantages of a good RDBMS.
Re:The Marketing Behind MongoDB by bulldoa on Tuesday August 29, 2017 @03:11PM (#15126824)
Would you recommend some resource for learning to setup postgres? As a new grad I often times find myself frustrated with setting up postgres or other rdbms so I usually opt for easier systems like amazon RDS, firebase or mongo.

Before docker became popular, I basically have to do something like [this](https://github.com/docker-library/postgres/blob/master/9.6/D...), which I don't think I have mastered till this day.

Re:The Marketing Behind MongoDB by brandur on Tuesday August 29, 2017 @06:34PM (#15128189)
> As a new grad I often times find myself frustrated with setting up postgres or other rdbms so I usually opt for easier systems like amazon RDS, firebase or mongo.

Yeah, I know exactly what you mean. Everything in Postgres is supremely well documented, but there's just so much documentation, that getting the basics down is pretty hard. I definitely went through a period of building muscle that was a tad frustrating to say the least.

Unfortunately, I don't know a particularly succinct resource that I recommend right now, but I will suggest taking a look at official docs. Maybe the section on managing databases in particular [1]. In general, you should just get good at using createdb, dropdb, and psql (which is mostly just generic SQL).

If you're on a Mac, installation and setup might looks as easy as:

    brew install postgresql
    brew services start postgresql
    createdb my-test-db
    psql my-test-db
    dropdb my-test-db
(There's obviously far more there than just that, but at a basic level, most of the commands are reasonably easy to use.)

[1] https://www.postgresql.org/docs/9.6/static/managing-database...

Re:The Marketing Behind MongoDB by tracker1 on Tuesday August 29, 2017 @08:03PM (#15128663)
Now setup 3-5 replica nodes with at least automatic master election and failover.
Re:The Marketing Behind MongoDB by qaq on Tuesday August 29, 2017 @10:32PM (#15129313)
and you need that because ...?
Re:The Marketing Behind MongoDB by closeparen on Wednesday August 30, 2017 @01:16AM (#15129870)
You shouldn't go down when one box or VPS falls over?
Re:The Marketing Behind MongoDB by oblio on Wednesday August 30, 2017 @04:26AM (#15130557)
For 99% of the situations you just need 2 boxes.
Re:The Marketing Behind MongoDB by mbesto on Wednesday August 30, 2017 @05:51AM (#15130872)
And if you do need more, there's commercial solutions:

https://aws.amazon.com/rds/postgresql/

https://www.citusdata.com/

Re:The Marketing Behind MongoDB by qaq on Thursday August 31, 2017 @04:45PM (#15143773)
RDS is very far from optimal for high load PG
Re:The Marketing Behind MongoDB by tracker1 on Wednesday August 30, 2017 @05:35PM (#15136257)
And, I've worked in a few that needed more... million+ simultaneous users, and over 380k requests per second, and 5 nines. There really are times you need more than an SQL server can keep up with.

Another, it was the shape of the data, all keyed off a single record, but ancillary tables with additional queries or joins meant over 30 joins, or 15 additional queries to render one resource, which fit into a single "nosql" record and mongo supported the additional indexes needed, it was a good fit for the use case.

I'm not advocating nosql for everything... I've seen a few that went mongo that would have been better served with sql. But that isn't to say that a document or column-store database is never a better fit from the start.

Re:The Marketing Behind MongoDB by qaq on Wednesday August 30, 2017 @07:16AM (#15131174)
You shouldn't and that is trivial to setup.
Re:The Marketing Behind MongoDB by dahdum on Tuesday August 29, 2017 @05:38PM (#15127806)
Docker works fine, but I prefer to run a vagrant machine with ansible to setup postgres + everything else for development. In production I just use RDS.

This is the ansible role I use: https://github.com/ANXS/postgresql

Re:The Marketing Behind MongoDB by bulldoa on Tuesday August 29, 2017 @06:25PM (#15128159)
do you guys have replication set up to your dev such that dev will have somewhat current info? If so how do you guys handle that replication?
Re:The Marketing Behind MongoDB by dahdum on Tuesday August 29, 2017 @07:16PM (#15128425)
Since we take nightly backups of the RDS instances and store them in S3, I just wrote some ansible commands to download/import into the local db.
Re:The Marketing Behind MongoDB by rmrfrmrf on Tuesday August 29, 2017 @03:52PM (#15127108)
> the startup overhead of an SQL system is a myth.

It's also a myth that devs choose NoSQL over SQL because "SQL is too hard."

Re:The Marketing Behind MongoDB by kafkaesq on Tuesday August 29, 2017 @08:49PM (#15128862)
It's also a myth that devs choose NoSQL over SQL because "SQL is too hard."

What I see more of is situations where the developers "know" SQL but still just don't want to sit down and figure out their data models.

It's like, the temptation to reach for the nearest short-cut -- whether that means over-reliance on ORMs (or hodgepodge 'data access layers'); or continually munging stuff at the application layer for nearly every operation; or... Mongo -- is always just too great.

Re:The Marketing Behind MongoDB by bryanrasmussen on Wednesday August 30, 2017 @05:19AM (#15130756)
figuring out the data model as you build can be beneficial, you realize what you need as you work with the thing and can add it on the fly, at a certain point of course you need to do a cleanup. Getting people to accept the cleanup requirement is the difficult part.
Re:The Marketing Behind MongoDB by tracker1 on Tuesday August 29, 2017 @08:02PM (#15128661)
Can't speak for anyone else... for me, it was the shape of the data I was interacting with was best represented as nested objects... think a classifieds site... primary record is the ad/entry, but different types of products will have differing ancillary data. That was my use case in first using MongoDB. Why, because otherwise there were up to 30+ joins in order to bring up all the data around any given classified entry.

Reshaping the data into a document store, with a handful of indexes made a lot of sense. Nearly effortless replication was also a nice feature, and needed for some extra endurance and scale. I'm pretty sad that RethinkDB didn't get farther, as their management interface and approach was a lot better than MongoDB.

To me it's about what does the data look like, what are the performance needs, and how much do you want to spend, cross-train or hire out expertise. Not to mention the lock-in options. There are lots of options, and it depends on the project and use. I've worked in enough applications where at least parts of the application needed to be moved off of SQL short of spending FAR more on licensing and/or servers that were insanely expensive for the need the value wasn't good.

Re:The Marketing Behind MongoDB by thehardsphere on Wednesday August 30, 2017 @11:38AM (#15133034)
Unfortunately, it's one of those myths that is prevalent enough because there are enough proponents of NoSQL who give it as their reason for doing it.

Kind of like how people also say they choose Node.js so they only have to use one language on the server and the client.

Re:The Marketing Behind MongoDB by api on Wednesday August 30, 2017 @10:37AM (#15132436)
The issue is absolutely not the overhead of schemas or managing them.

The issue is the overhead of systems administration when clustering. Clustering is a requirement to support high availability, and Postgres clustering is a real bear to set up. Mongo clustering is easy.

Postgres and most other SQL databases hail from the era when things ran on a "box." They're designed that way. Clustering is a bolt-on. It's that more than queries or schemas that drive NoSQL.

It would have been better to fork Postgres and create a cleaned-up easy to cluster version minus the obnoxious arcana, but that's no fun. It's more fun to reinvent the wheel.

Edit: We migrated from Postgres to RethinkDB for this reason and are mostly happy with it. We miss the data integrity guards of SQL but getting clustering without a full time DBA and Raft consensus fail-over without intervention is a worthy trade off. We still use Postgres for back-end warehousing and analytics but those are not live systems so they can live on a "box."

Re:The Marketing Behind MongoDB by antoaravinth on Wednesday August 30, 2017 @02:53AM (#15130157)
>I've used Postgres and Mongo pretty extensively, and for any reasonably seasoned developer, the startup overhead of an SQL system is a myth. There may upfront cost to learning how an RDMS and SQL work in the first place, but once you're familiar with them, they'll be faster than Mongo on any new project.

Exactly, I was very new to the RDBMS world and always I have preferred to go for MongoDB. Once in my side project, I thought of learning RDBMS and used Postgresql. Once I learnt how transactions, joins works then RDBMS was totally an easy topic.

I have used Jooq instead of ORM, which again helped me to learn the queries and underlying system of Postgresql!

Re:The Marketing Behind MongoDB by athenot on Tuesday August 29, 2017 @03:33PM (#15126975)
In the words of one of my former DBAs: "MySQL makes a pretty good noSQL database".

That is to say you can do a quick and dirty key-value store in MySQL and it will be plenty fast. When you're ready to start normalizing things, just start pulling values out into their own columns and/or tables.

Conversely, when starting a product, don't worry about deeply normalized databases: use a relational DB as a glorified key-value store, then normalize things as you go. You'll get the best of both worlds.

Re:The Marketing Behind MongoDB by sushisource on Tuesday August 29, 2017 @04:42PM (#15127447)
Yep. I've done the same thing by using JSONB in postgres.
Re:The Marketing Behind MongoDB by tracker1 on Tuesday August 29, 2017 @08:05PM (#15128672)
I've done similar for most SQL databases having either an xml, or json column to store ancillary information that didn't need to be accessed or indexed in ways that aren't programmatic.

For example, if you support Amazon Payments, Paypal and a payment gateway, the shape of the data for the transaction is different, and rarely looked at, why not shove it into a bigtext/blob field as xml/json.

Re:The Marketing Behind MongoDB by manigandham on Tuesday August 29, 2017 @07:06PM (#15128354)
Yes, relational databases are easy deal with. A single machine with proper backups can handle data needs for most companies. The explosion of managed service providers makes this even cheaper, faster and easier to start with.

ORMs can do 95% of most project data access needs and a little bit of SQL knowledge allows for easily creating and editing tables - or use the numerous GUI tools out there.

If anything, SQL actually lets you be more agile by allowing for quick queries to fix data or do a complicated 1-time report without having to write up and deploy code. This probably has more to do with the SQL language itself which is why it's surprising that MongoDB never tried to implement any of it.

Re:The Marketing Behind MongoDB by tracker1 on Tuesday August 29, 2017 @08:08PM (#15128693)
But the cost of a server big enough to handle a million simultaneous users can be cost prohibitive if you can get away with a few smaller nodes and something that scales horizontally... not to mention failover/redundancy which can be downright expensive and/or painful in many SQL variants.
Re:The Marketing Behind MongoDB by manigandham on Wednesday August 30, 2017 @12:36AM (#15129757)
That's the whole point - worry about that when you actually have a million simultaneous users, which is a huge number. There are plenty of options like reading from replicas, using a caching layer, or moving specific data to another datastore - all work that's better done when you actually have success and the resources to understand your data access patterns and implement a smart solution when it's needed.

Basic replication has been part of every major relational database for decades. Where is it expensive or painful? Do you really need 100% uptime? There's a vast ecosystem for relational databases that can help with all of this and now you have CitusDB, MemSQL and CockroachDB if you need built-in next-gen scaling and availability.

Re:The Marketing Behind MongoDB by qaq on Tuesday August 29, 2017 @10:45PM (#15129359)
The cost of dev time having to reimplement some of the features available in RDBMS on top of NoSQL stores can be NxTimes the cost of of that box.
Re:The Marketing Behind MongoDB by rmrfrmrf on Tuesday August 29, 2017 @03:50PM (#15127086)
This is kind of a ridiculous statement given the massive popularity of NoSQL DBs like Cassandra, Redis, and Lucene.

Not to mention what you described seems like a perfectly reasonable architectural evolution. Are we all really expecting to make a single decision on a DB for the entirety of an app's life? It makes a lot of sense to use a flexible db schema while the data takes shape in an app's infancy, then solidify things with a more structured database schema further down the line. If you're abstracting things away in ORMs anyway, it might not even be that painful if you're just switching out DB drivers.

Re:The Marketing Behind MongoDB by jimktrains2 on Tuesday August 29, 2017 @05:26PM (#15127736)
Most people don't use Redis and Lucene as general-purpose data stores; they're normally used for specific purposes.
Re:The Marketing Behind MongoDB by thehardsphere on Wednesday August 30, 2017 @11:44AM (#15133089)
I would go so far to say that nobody should ever use Lucene as a data store. It's very easy for it to get corrupted because it was explictly designed to support fast full-text searches at the expense of persistence (because your data is supposed to be persisted elsewhere).
Re:The Marketing Behind MongoDB by camus2 on Tuesday August 29, 2017 @06:03PM (#15128000)
Nobody uses Redis like a database and expects persistence of data. Redis is usually used in the context of a message queue.

Lucene isn't a database, it's a full text search engine.

Cassandra has a query language which is very close to SQL, so it's not really NoSQL.

Re:The Marketing Behind MongoDB by jimktrains2 on Tuesday August 29, 2017 @10:28PM (#15129300)
> Redis is usually used in the context of a message queue.

Or caches, or simple data stores for things like counts.

Re:The Marketing Behind MongoDB by manigandham on Wednesday August 30, 2017 @10:32AM (#15132405)
> Nobody uses Redis like a database and expects persistence of data.

Plenty of people do, it's got just as good persistence and backups as anything else. It's very versatile and used for lots of situations, with message queues just being one option (although not a great fit until it gets a stream data type).

Re:The Marketing Behind MongoDB by joshlemer on Wednesday August 30, 2017 @11:55PM (#15138054)
>Nobody uses Redis like a database and expects persistence of data.

You'd be surprised. I know of at least one (extremely) large company that does / has done this.

Re:The Marketing Behind MongoDB by richardknop on Wednesday August 30, 2017 @11:34AM (#15132989)
Redis is either used as pub sub / message queue but also often it is used for caching similar to memcached.
Re:The Marketing Behind MongoDB by thehardsphere on Wednesday August 30, 2017 @11:41AM (#15133061)
Caching does not have the same expectations for persistance that a database has. If data in Redis gets blown away, I don't really care. If data in my RDBMS or NoSQLDB gets blown away, it's a huge problem.
Re:The Marketing Behind MongoDB by richardknop on Wednesday August 30, 2017 @11:46AM (#15133112)
Yes, I agree. Most people I know would use RDBMS such as Postgres to store bulk of business data (customer accounts, transactions, receipts etc) and then use complementary systems for other purposes (so could be Memcache/Redis for caching, ElasticSearch for logs etc).

It's not a false dichotomy of using either Postgres or Redis, you can use them alongside each other, they have different uses.

Re:The Marketing Behind MongoDB by andy_ppp on Tuesday August 29, 2017 @07:06PM (#15128352)
Canssandra is much more difficult than an SQL database to build functionality in, but can scale linearly with the number of nodes added. I quite enjoyed it’s deliberately imposed limitations to help you write a scalable data layer.
Re:The Marketing Behind MongoDB by randomdata on Tuesday August 29, 2017 @03:42PM (#15127031)
0. Finally, an API that actually attempts to fit modern programming paradigms!

I'm not sure the unstructured nature of many NoSQL databases was even the selling feature so much as the NoSQL part. People were growing tired of dealing with lacklustre ORMs and poor integration with their programming tools and were desperate, even in light of the shortcomings, to try something new.

It is funny that we have a thousand and one different general purpose programming languages that try to solve the problems with general purpose computing, but SQL dialects (with slight variation here and there), have very little competition, despite all the problems with the SQL language.

A NoSQL relational database could have potentially hit the sweet spot, but Mongo and similar databases came first and caused enough trouble, as you have described, that I feel like they've killed the momentum in looking for better solutions to access database data (relational ones included). Nowadays many of those people are just happy to have a relational database again, even that means accepting SQL.

Re:The Marketing Behind MongoDB by jdmichal on Tuesday August 29, 2017 @04:29PM (#15127357)
> It is funny that we have a thousand and one different general purpose programming languages that try to solve the problems with general purpose computing, but SQL dialects (with slight variation here and there), have very little competition, despite all the problems with the SQL language.

This is kinda like complaining that there's only one implementation of lambda calculus. Yes, SQL is not directly a relational algebra, but it's pretty close. (Also, tables are not relations and SQL deals with that.)

Re:The Marketing Behind MongoDB by randomdata on Tuesday August 29, 2017 @05:02PM (#15127576)
> This is kinda like complaining that there's only one implementation of lambda calculus. Yes, SQL is not directly a relational algebra, but it's pretty close.

And yet someone proficient in SQL will not necessarily be able to read a traditionally-written lambda calculus expression as the languages differ significantly (and not just because of tables), so I think this proves my point that there are different, and hopefully better, ways to express the interaction with the database. There is no reason why the protocol has to be exactly as SQL is. There are theoretically an infinite number of ways to communicate the same intent to a RDBMS.

For instance, SQL does not do composability well. There is no fundamental reason why you should not be able to write query fragments, simplifying the way many queries are constructed and avoiding the huge mess that monolithic, hard to debug, queries often become. It is simply that SQL does not allow for it. A better API could improve on this significantly, without losing any features a relational database provides. You don't have to throw all database theory out the window just to change the API away from one that was designed 40 years ago.

Re:The Marketing Behind MongoDB by jdmichal on Tuesday August 29, 2017 @06:04PM (#15128017)
> And yet someone proficient in SQL will not necessarily be able to read a traditionally-written lambda calculus expression as the languages differ significantly (and not just because of tables), so I think this proves my point that there are different, and hopefully better, ways to express the interaction with the database. There is no reason why the protocol has to be exactly as SQL is. There are theoretically an infinite number of ways to communicate the same intent to a RDBMS.

First, lambda calculus isn't a language -- at least, not in the typical sense. It's a mathematical logic construct. It's more akin to geometry than to a programming language. Same applies to relational algebra.

And of course lambda calculus and relational algebra differ significantly... Their domains are completely disjoint. I was not comparing them in this way. I'll restate my premise:

If we consider SQL a mapping of relational algebra operators to SQL operators [0], then any other language which also maps to relational algebra will not be significantly different. Thus, complaining about SQL being a singularity is kinda like complaining that relational algebra, lambda calculus, or any other mathematical constructs are singularities.

> For instance, SQL does not do composability well. There is no fundamental reason why you should not be able to write query fragments, simplifying the way many queries are constructed and avoiding the huge mess that monolithic, hard to debug, queries often become. It is simply that SQL does not allow for it. A better API could improve on this significantly, without losing any features a relational database provides.

You can compose SQL with views. Non-materialized views act as pure SQL composition. Materialized views allow precalculation of subexpressions.

Most of the big databases also support common table expressions, which allow for decomposition within a single query.

https://en.wikipedia.org/wiki/Hierarchical_and_recursive_que...

> You don't have to throw all database theory out the window just to change the API away from one that was designed 40 years ago.

Of course not. The problem is that any new SQL would just be a new mapping for the same underlying relational algebra. "Welcome to the new SQL, same as the old SQL."

[0] It's not, because tables aren't relations. However, the operators are similar enough that parallels are easily drawn.

Re:The Marketing Behind MongoDB by randomdata on Thursday August 31, 2017 @12:09PM (#15141292)
> First, lambda calculus isn't a language -- at least, not in the typical sense.

There are common ways to express lambda calculus expressions in order for people to communicate those expressions with each other, which is very much a language. After all, we are talking about the language of SQL here, not the mathematical concepts that lie beneath the language. Don't confuse what SQL is able to achieve with the way SQL is written. We are discussing only the latter.

> The problem is that any new SQL would just be a new mapping for the same underlying relational algebra.

And general purpose programming languages map to the same underlying turing machine. That hasn't stopped us from coming up with variations in language to improve the developer experience, even if they are all quite similar at the end of the day. Let's face it, if you can program in one general purpose programming language, you can program in them all. There are no real differences between them. The only thing to learn when switching languages is the grammar and idioms of the language. The underlying mathematical theory never changes.

You could just as easily say "Welcome to the new C, same as the old C." And yet, the new C has proven to be quite valuable when it comes to developer productivity and happiness. There is very much room for the same in SQL land, if someone was ever motivated enough to look for it. I maintain that NoSQL becoming synonymous with unstructured key-value/document data stores has killed what motivation there may have been though.

Re:The Marketing Behind MongoDB by rgo on Tuesday August 29, 2017 @03:38PM (#15127007)
Here's my personal version of the story:

With SQL DBs (Oracle, SQLServer and MySQL):

1. SQL database migrations where killing us. Going back and forward in a dev environment was impossible. No hot deploy in production.

2. Could not work well with application user-defined fields: adding columns adhoc to the database, indexing them, normalizing and denormalizing, performance issues, everything was a problem.

3. Blobs holding logging data got unmanageable quickly.

4. Joins where very hard to optimize even though the team had a lot of DBA experience fine tuning databases.

5. Had to build a very complex architecture around the database for a product that was not that complex: cache, search, database, blob store, distributed, etc.

And with all our 1990s and 2000s previous experiences in data warehousing, business intelligence and DB optimization tools, we were still wasting valuable time with SQL design, indexing, query planning and parameter optimization. So we gave MongoDB a try. First as a cache. Later as the only DB.

Our journey:

1. Heard about Mongo. Tried the DB. The driver worked great. To me that's the number one "marketing antics behind MongoDB": their strategy creating drivers and supporting the programmer community.

2. Understood what NoSQL meant and forgot about joins altogether.

3. Understood what NoSQL meant and built transactions into atomic documents.

4. Understood what NoSQL meant and stopped relying on the database for type, primary and foreign key constraints, default values, triggers (argh!), stored procedures (2x argh!), etc.

5. Simplified the architecture with integrated search, queue and cache. Less moving parts = joy.

6. Result: very low maintenance, easy install, configuration, replication and migrations. 99.999% availability.

7. Bonus: we even implemented a very high frequency, atomic distributed semaphore system with a FIFO queue that reaps zombies using Mongo built-in networking features.

So we've reduced DB-related issues by an order of magnitude. How? I think because NoSQL is a way of saying the DB should not be magically answering random queries. A database should be a data store, period -- just store and retrieve data the way the app needs it. By focusing our energies on getting the data right as documents for a document store meant data flows as objects from code in and out of Mongo.

I believe people underestimate how important (and productive) it is to keep the same data structures flowing between the UI (JSON), server (Object/Hash/Dictionary) and DB (document). It makes code easier to read and more resillient to errors.

But SQL DBs come with a convenience layer bolted on to run random user queries with things like OUTER joins and GROUP BYs. For that we need to flatten data into tables, which clashes with typically how data flows in an app.

SQL DBs however are great as the single source of truth for data: a schema can be laid out and enforced independently of code, so it's safely guarded from programmers breaking it. Business sets up a SQL DB so that their reporting people can query data on demand while consultants with zero knowledge of the business can write code limited by constraints managed by DBAs. SQL is even taught at business schools, which is revealing of who its target audience actually is.

Bottom-line: SQL and schema enforcing are end-user features we did not need to build our tool. On the other hand, every single MongoDB feature is something we need and use profusely.

Re:The Marketing Behind MongoDB by DaiPlusPlus on Tuesday August 29, 2017 @04:08PM (#15127212)
> 2. Understood what NoSQL meant and forgot about joins altogether.

How would you represent a simple invoicing system in MongoDB (e.g. Customers + Products + Orders + OrderLineItems )? NoSQL-for-everything advocates posit two solutions: either denormalize the data by embedding Customer information within an Order document, which also contains an array of OrderLineItems, or use a UUID as a kind-of foreign key and maintain separate relationships. Both approaches have serious problems (data-duplication and inevitable inconsistency in the first, and lack of referential integrity in the second, besides ending-up abusing a NoSQL database as an RDBMS). Is there a better way? Or would you agree that certain classes of problems are best left to RDBMS' domain?

Re:The Marketing Behind MongoDB by rmrfrmrf on Tuesday August 29, 2017 @05:08PM (#15127614)
The example you've used (invoices) is actually quite instructive for demonstrating the benefits of a "document store." An invoice, historically, was a literal printed piece of paper. Invoices are actually really annoying to implement in an RDBMS because of so-called "referential integrity" -- an invoice should be a "snapshot in time" of everything that happened when the order was processed, so ideally, when a user views their invoices from the past 2 years, they look the same every time.

Except, oops, your user got married and moved, now your precious "referential integrity" means jack because the generated invoice is flat-out wrong. Product removed from the store? Too bad, needs to stay in the database forever for historical purposes. Prices need to change? Better design the database to handle snapshots of every product state.

If you were implementing this in MongoDB, you'd probably store a UUID and the flattened data at the time of invoice generation, that way you can still query on ids AND not deal with the headache of having a combinatorial explosion of data in your RDBMS.

Re:The Marketing Behind MongoDB by icedchai on Tuesday August 29, 2017 @05:25PM (#15127732)
You would solve this in a RDBMS the same way: de-normalize when you're saving the invoice (example: a line items table with snap shot of current item price, description, etc.)
Re:The Marketing Behind MongoDB by rmrfrmrf on Tuesday August 29, 2017 @08:46PM (#15128851)
Yes, which suggests that the "serious problems" mentioned by the grandparent aren't serious (or problems) at all.
Re:The Marketing Behind MongoDB by purerandomness on Thursday August 31, 2017 @11:39AM (#15141054)
In Postgres, you'd simply have a table with a JSON column for the snapshot-in-type contract.

You can then select fields from that JSON for invoices, reports, etc with the arrow operator:

https://www.postgresql.org/docs/9.6/static/functions-json.ht...

Re:The Marketing Behind MongoDB by wvenable on Tuesday August 29, 2017 @05:41PM (#15127832)
With SQL you can denormalize all that (and should) to create that snapshot. But with NoSQL you can't normalize and get back a way to quickly query the number of products sold per month over the last 5 years.
Re:The Marketing Behind MongoDB by chrisco255 on Tuesday August 29, 2017 @07:15PM (#15128420)
Yes, this is possible with Aggregation and MapReduce: https://docs.mongodb.com/manual/aggregation/
Re:The Marketing Behind MongoDB by wvenable on Tuesday August 29, 2017 @08:12PM (#15128709)
For relative values of "quickly".
Re:The Marketing Behind MongoDB by imtringued on Tuesday August 29, 2017 @04:45PM (#15127465)
Instead of nebulous terms like NoSQL you should instead just look at the damn features because these concepts are orthogonal. MongoDB has transaction isolation on the document level instead of the database level. If you can store everything in a single document then it doesn't matter. If you can't then use a database that supports database level transactions. It doesn't matter if it's a NoSQL or RDBMS database.

I feel a lot of people know that typical nosql databases (without database level transactions) are not suitable for their problem but they don't know why and then just think NoSQL is always bad and RDBMS are always better because the NoSQL databases are intended to be used for different problems.

Re:The Marketing Behind MongoDB by kreetx on Tuesday August 29, 2017 @04:39PM (#15127427)
Not the original commentor, but there are some valid cases for NoSQL: some people use it for storing massive amounts of web crawling data. But the thing here is that it's throw-away'ish, and in that case it's often not worth it to add structure (even though there pretty much is structure in everythig you look at long enough).

But I do think having any data consisting of, say, items, orders, users, payment in MongoDB is very much a bad idea. Been there.

Re:The Marketing Behind MongoDB by mdpopescu on Wednesday August 30, 2017 @01:07AM (#15129843)
> I think because NoSQL is a way of saying the DB should not be magically answering random queries.

The reason this is wrong is something that Codd et al learned a while ago: the data is MORE IMPORTANT than the application. Applications change and/or become obsolete; the data doesn't. You will still need to query the same database 50 years from now, but you likely won't have the same application to do it with. That means that everything that is important to the data (schema, constraints and so on) needs to stay with the data.

Re:The Marketing Behind MongoDB by Something1234 on Tuesday August 29, 2017 @04:28PM (#15127351)
What was your tool?
Re:The Marketing Behind MongoDB by mosselman on Wednesday August 30, 2017 @05:48AM (#15130862)
What nonsense. I have never considered using postgres a hindrance due to migrations or something of the like. If anything, migrations and having a consistent database makes it easier to reason about what you code should do without having to take care of missing pieces of information in noSQL objects that have become stale because of 'speed of iteration'.
Re:The Marketing Behind MongoDB by ibejoeb on Wednesday August 30, 2017 @10:58AM (#15132637)
I really wonder what all of these claimants do. Do they change their data models and corresponding procedures, and then just leave all of the stale, nonconforming data in there? How is that working out?
Re:The Marketing Behind MongoDB by vram22 on Tuesday August 29, 2017 @08:05PM (#15128671)
>I think the most interesting question though is would they be able to get MVP and initial customers that set off this if they were moving (slightly) slower due to SQL and slight overhead that comes with?

Why the heck not? People (startups and others) were using RDBMSs and SQL for years before NoSQL was a thing.

Also, if slight delay is a thing, how about the impact of your point 7 on customers?

Re:The Marketing Behind MongoDB by TruffleMuffin on Wednesday August 30, 2017 @08:51AM (#15131645)
I have been using Mongo for several years now on a fresh build that follows basically the outline you have here. The fresh build was triggered in part because of the cost (development time and infrastructure) in using Microsoft SQL Server. The only difference is that I got to step 5, realised I had two very different reporting requirements. One that Mongo works well with (real time aggregated data), and one which SQL works well with (high granularity, long time period, historical stuff). So, I use Mongo for the first requirement, and SQL for the later.

I see many posts like this, and indeed many replies like it also (your thread currently). Curious, why do they always focus on have only one database to fit all their needs? Is my situation special, can't you just make a decision about what tool is best for the job and use it? Is there some eventuality where having databases specific to needs results in disaster that I am just not aware of?

Re:The Marketing Behind MongoDB by moeamaya on Tuesday August 29, 2017 @06:19PM (#15128126)
JSONB in pgSQL fills in so nicely for steps 1-4. Once you need columns and relationships, the relational store is already there!
Re:The Marketing Behind MongoDB by jwilliams on Tuesday August 29, 2017 @02:55PM (#15126710)
I largely agree with your journey and your premise.

However, I would add that I've seen just as many beast-of-a-thing SQL databases as NoSQL ones. In fact, anything I've seen at sufficient scale has lots of wrangling with their database.

While your step 7 usually ends up with a better tech result, it's done at a time when the organization knows the problem domain much (much) better. They have real users. You'd probably get a similar result going from one SQL database to another.

Re:The Marketing Behind MongoDB by wolco on Tuesday August 29, 2017 @02:44PM (#15126620)
I'm doing the same for another project. Reporting with $wind can be limiting

Schemaless db is great feature. Joins can be done through an orm like Laravel's eloquent which allows you to join mysql to mongo using a cross-join table on the mysql side.

Adding columns in an mpv product is trivial. Getting a stream of unstructured data and being able to save the different data to a document that is part of the same collection is an amazing feature that is needed still.

Re:The Marketing Behind MongoDB by DaiPlusPlus on Tuesday August 29, 2017 @04:01PM (#15127173)
Adding a non-NULL column to an existing table with millions of rows, and then updating the entity/model objects upwards towards the UI is non-trivial - I find myself "queuing up" planned database schema changes and applying them in a batch.

What I'd love is a traditional database system with the concept of "cheap columns" that store "just data" scalar values, which can be easily created, dropped and manipulated - they're addressable so they can be used in queries and statistics/analytics, but the values themselves won't need indexes or referential-integrity - and for an ORM and Scaffolding system to support them - that way I won't pull out my hair when the client says one day "let's go from a single column for the customer's name to having separate title/first/middle/last/suffix columns"... and back again the week after.

Re:The Marketing Behind MongoDB by jdmichal on Tuesday August 29, 2017 @04:36PM (#15127411)
So something like PostgreSQL `json` and `jsonb` types? You can select them out into columns, so they should play nicely with ORMs.

https://www.postgresql.org/docs/current/static/datatype-json...

Re:The Marketing Behind MongoDB by solidsnack9000 on Wednesday August 30, 2017 @02:02AM (#15129998)
It seems like the "column families" of Cassandra would appeal to you.
Re:The Marketing Behind MongoDB by hacktothefuture on Wednesday August 30, 2017 @09:06AM (#15131757)
This! 100% this. I loved phases 1-4. Then 5 hit and horror show. I realized quickly that I was building kludgy extraction routines to get the reports/metrics I needed. Once the horrible process became unmanageable I would have gladly paid the longer development/iteration time on phases 1-4 in order to have a robust and resilient system which allows proper reporting.
Re:The Marketing Behind MongoDB by konschubert on Tuesday August 29, 2017 @04:02PM (#15127174)
My feeling is that the people who flock to NoSQL have never used a good ORM such as Django to abstract away the raw SQL queries. A good ORM gives you the simplicity of NoSQL plus all the power of a relational database.
Re:The Marketing Behind MongoDB by thehardsphere on Wednesday August 30, 2017 @11:52AM (#15133164)
I think that is only true of a subset of NoSQL users.

And even if it is true that Django's ORM is very nice, there are tradeoffs to using an ORM that mean you don't get "all the power" of whatever database you are using. Maybe all the power you care about; for instance, power in expressiveness and easy cross platform support, at the expense of performance optimizations that aren't important to you. But it's still a tradeoff.

Re:The Marketing Behind MongoDB by konschubert on Wednesday August 30, 2017 @01:16PM (#15133969)
You're right.