News for hackers. Stuff that matters.
Re:Startup Engineers and Our Mistakes with MongoDB by MikeKusold on Wednesday July 19, 2017 @12:42PM (#14805608)
> By far the most consistent mistake was choosing a non-relational database, when your data was strongly relational. Mongoose's ODM made this mistake surprisingly easy to make, which led to issues down the line.

This mirrors my experiences with Mongo as well. The vast majority of data is relational. Mongoose allows people to make the mistake of structuring their data relationally. But all you are doing is pushing all the joins to the web server (which in Node land is single threaded and compounds the mistake of choosing NoSQL).

There may be some microservices that Mongo is great fit for, but it should not be your core data store.

Re:Startup Engineers and Our Mistakes with MongoDB by scarface74 on Wednesday July 19, 2017 @01:56PM (#14806359)
But all you are doing is pushing all the joins to the web server (which in Node land is single threaded and compounds the mistake of choosing NoSQL).

Why is pushing the joins "to the web" inherently bad? It's a lot easier to scale web servers than database servers.

Re:Startup Engineers and Our Mistakes with MongoDB by nulagrithom on Wednesday July 19, 2017 @03:39PM (#14807248)
Simplified, a JOIN on an SQL server will take advantage of its indexes to grab the data from the disk that matches the specified criteria. SQL databases are built for this sort of thing, and they're really freaking good at it.

Sending that same JOIN to the web server means, at the least, shipping everything from both tables matching the criteria and matching up those two potentially huge blobs of data with each other on the web server. Doing this in an even partially efficient manner will split your query logic between DB queries and other code. Also, it's not unlikely that you'll be doing this JOIN operation in a language that maybe isn't great at optimizing that sort of CPU intense behavior on its own, so you're probably going to need to drill in and optimize on your own (vs letting SQL do its magic). Plus, we're going to need to store all this data in memory while we match it. So now our web servers, while more easily scaled than a DB, are actually pretty damn big boxes with decent memory and CPU, and that's expensive to scale.

Re:Startup Engineers and Our Mistakes with MongoDB by scarface74 on Wednesday July 19, 2017 @06:02PM (#14808401)
If you are doing a lot of joins in Mongo - you're doing it wrong. The whole purpose of a document DB is storing your model with the related data as one document.
Re:Startup Engineers and Our Mistakes with MongoDB by nulagrithom on Wednesday July 19, 2017 @07:01PM (#14808753)
If we dump everything we might JOIN on to a single document then in many cases we're going to end up with a ridiculously nested document, and/or duplicate data everywhere. Let's try mapping out the project I'm working on right now.

We've got moves, and each move has multiple stops:

  move: {
    shipper: string,
    consignee: string,
    stops: [{
      address: string,
      appt: datetime,
    }] 
  }
So far this looks great. I like this. Yay MongoDB!

But I need to add a driver to each stop... Where do I put the driver? If I put him under the stop I'm going to duplicate his data all over the place. Do I put the move under the driver instead?

  driver: {
    name: string,
    id: string,
    moves: [{
      shipper: string,
      consignee: string,
      stops: [{
        address: string,
        appt: datetime,
    }]
  }
Ok, not bad. We've got an array of objects under an array of objects but I'm still perfectly comfortable.

Now I need to get a terminal-based look at all the drivers, so I know what every driver is doing in a particular city, also drivers can actually have multiple trucks too:

  terminal: {
    city: string,
    drivers: [{
      name: string,
      id: string,
      truck: [{
        plate: string,
        moves: [{
          shipper: string,
          consignee: string,
          stops: [{
            address: string,
            appt: datetime,
          }]
        }]
      }]
    }]
  }
That's getting a touch nested but... Drivers actually work for vendors, and vendors might work across multiple terminals. Do we duplicate the vendor across terminals, or the terminal across vendors? Also even the driver/truck relationship is many-to-many... Do we duplicate the truck info? What happens when we need to update a truck? Do we attempt to update across all drivers? Do we duplicate that data too?

  terminal: {
    city: string,
    vendor: [{ // many-to-many! duplicate vendor data?
      id: string,
      drivers: [{
        name: string,
        id: string,
        truck: [{ // many-to-many! duplicate truck data?
          plate: string,
          moves: [{
            shipper: string,
            consignee: string,
            stops: [{
              address: string,
              appt: datetime,
            }]
          }]
        }]
      }]
    }]
  }
Ok, this schema is starting to look like shit and it's only going to get worse. The shipper and consignee should actually be unique objects as well, and we might want to update the contact info on them. Do we duplicate that data and update across all moves?

Also we want to cross-reference the shipper, consignee, billing, and ordering entities for market analysis (the consignee isn't always the customer, and the customer doesn't even always get the bill!) Do we duplicate that information across moves? Do we duplicate moves across entities? More many-to-many relationships...

Maybe I'm just not creative enough to use MongoDB, but I'm really starting to feel like this is a square peg in a round hole.

Re:Startup Engineers and Our Mistakes with MongoDB by scarface74 on Wednesday July 19, 2017 @11:08PM (#14809790)
This isn't rocket scientist and people far smarter than us have already figured this stuff out.

https://martinfowler.com/bliki/DDD_Aggregate.html

If you look at this from a Domain Driven Design standpoint, you should model your business and think about your aggregate roots and each collection should be its own aggregate root, data access should be handled by one class/microservice for each aggregate root.

If you modeled your Domain correctly that wouldn't be an issue.

On the other hand, not modeling your domain correctly first would lead to an ungodly, untestable, tangle of stored procedures and 10 way joins with A relational database.

The Mongo docs goes into best practices which mirrors basically the concept of an aggregate root.

https://www.mongodb.com/blog/post/thinking-documents-part-2

Re:Startup Engineers and Our Mistakes with MongoDB by nulagrithom on Wednesday July 19, 2017 @11:44PM (#14809883)
The MongoDB document linked says "Referencing should be used to represent complex many-to-many relationships" ... "References are usually implemented by saving the _id field of one document in the related document as a reference. A second query is then executed by the application to return the referenced data."

So... It's like a join, but with two queries instead of one, and you push the join data to the webserver, which brings us back to the start of this comment thread...

Re:Startup Engineers and Our Mistakes with MongoDB by scarface74 on Thursday July 20, 2017 @07:24AM (#14811494)
From your scenarios and what your asking to get from your data (yes I understand your problem domain, I wrote mobile software for field service workers in a previous life), you're not doing a lot of complex two way joins where you need to get all of the data from table A and match it to all of the data from table B. You're starting with a known entity like a terminal with references to other known entities. In your scenarios all your DB is doing is searching tables based on indexes and creating a data set in memory. If you did the same searches one table at a time starting from your known root - the terminal you are looking for and query individual tables based on foreign references, the only performance difference would be the slight overhead of multiple DB calls. You wouldn't even be getting unnecessary data back.

Even if I were using a relational database, I would still separate out my different domains (drivers, terminals, stops, etc.) into different classes/microservices and join the different entities in memory.

This would allow different developers, teams etc. to focus on separate business objects and would allow you to start off with a monolithic code base/app and separate the app out to microservice later without intertwined logic and to mix and match storage based on what makes sense.

I never said always embed everything into one document. I said think through your aggregate roots to decide how to design your apps and your document model.

I've designed this way for the last 10 years - 9 using RDMSs. I don't do complex 10 way joins for apps. I separate out business domain logically and let each module be responsible for its own domain and then have an overarching orchestrating class that combines the various objects when needed.

Re:Startup Engineers and Our Mistakes with MongoDB by nulagrithom on Thursday July 20, 2017 @11:20AM (#14813235)
> I don't do complex 10 way joins for apps.

That hits home. There's a lot of this in the legacy code base, and yeah, it's painful.

Interesting approach. Thanks for the discussion. I'll have to ponder this a while.

Re:Startup Engineers and Our Mistakes with MongoDB by bdcravens on Wednesday July 19, 2017 @02:22PM (#14806591)
Joins are typically a tiny intersection of 2 data sets. Multiply that by the number of web requests. Moreover, a query planner is far more optimized than the code you'd write on your web server.
Re:Startup Engineers and Our Mistakes with MongoDB by kazen44 on Wednesday July 19, 2017 @03:21PM (#14807080)
also, scaling databases is not terribly difficult, it is just very "clumsy".

Still, having a proper transaction manager and query planner will give you far better performance compared to doing it on webservers.

Also, aren't bsically moving the complexity of the database to the webserver? Which makes your webserver far more complex and harder to scale in comparison. Especially if you need ACID. (which is a must if you are doing anything with data storage imo).

Re:Startup Engineers and Our Mistakes with MongoDB by scarface74 on Wednesday July 19, 2017 @06:07PM (#14808449)
You don't always need ACID. Sometimes eventual consistency is good enough. How would it make the webserver itself more complex? The application itself should just be able to run on n servers.
Re:Startup Engineers and Our Mistakes with MongoDB by mrtksn on Wednesday July 19, 2017 @12:56PM (#14805731)
If data needs to be processed, an option would be to use MongoDB for collecting the data in bulk and later decide how you need to structure it for your needs.

When it comes to "read" the data, you read it only from processed database that can be anything.

Since you can do your processing completely independently of your web server, your are not necessarily pushing your computational load from the DB to the web server.

Re:Startup Engineers and Our Mistakes with MongoDB by flavio81 on Wednesday July 19, 2017 @01:02PM (#14805792)
> If data needs to be processed, an option would be to use MongoDB for collecting the data in bulk and later decide how you need to structure it for your needs.

OR, i can open a file stream, serialize my data to JSON, entity by entity, and dump all to a file.

The good old file.

Re:Startup Engineers and Our Mistakes with MongoDB by threeseed on Wednesday July 19, 2017 @01:37PM (#14806142)
Because we want to do partial updates, searches, indexing etc.

Your position can be applied to all databases. Why not abandon them all and just use CSV ?

Re:Startup Engineers and Our Mistakes with MongoDB by bdcravens on Wednesday July 19, 2017 @02:24PM (#14806600)
You could always do that, dump the data to s3, and use Athena :-)

https://aws.amazon.com/athena/

Re:Startup Engineers and Our Mistakes with MongoDB by pritambaral on Wednesday July 19, 2017 @04:03PM (#14807432)
And then be locked-in to Amazon. From ubiquitous files to ... a single service provider.
Re:Startup Engineers and Our Mistakes with MongoDB by mrtksn on Wednesday July 19, 2017 @01:05PM (#14805831)
replied to the same question here: https://news.ycombinator.com/item?id=14805812
Re:Startup Engineers and Our Mistakes with MongoDB by aidos on Wednesday July 19, 2017 @01:10PM (#14805876)
Then again, if you're going to push into Postgres you might as well store the temp unstructured records there too
Re:Startup Engineers and Our Mistakes with MongoDB by mrtksn on Wednesday July 19, 2017 @01:23PM (#14805989)
Well, that's also an option :) But I believe this is a relatively new option.

Personally, I never got into the hype with the NoSQL and for me the only use is as a convenient API for storing JSON files during prototyping so that I can later decide how my data should be structured.

Definitely tech debt for the future but fast iterated design that needs the database stuff fixed has more chance for success than the rigid design with the perfect data architecture.

Re:Startup Engineers and Our Mistakes with MongoDB by emodendroket on Wednesday July 19, 2017 @12:58PM (#14805755)
It that point it's harder for me to understand why I'm not just writing out JSON files or something.
Re:Startup Engineers and Our Mistakes with MongoDB by mrtksn on Wednesday July 19, 2017 @01:03PM (#14805812)
You definitely can but MongoDB provides convenience over storing and managing JSON files on you filesystem by your own efforts.
Re:Startup Engineers and Our Mistakes with MongoDB by lowbloodsugar on Wednesday July 19, 2017 @01:42PM (#14806209)
No, I really can't think of any situation where installing, managing, updating, maintaining, and crying over a mongodb cluster would be easier for a flat-file datastore than say, a file system, or S3. In fact if one is on AWS, then nothing beats just dumping them in S3 and processing them as necessary. If I'm a startup and I really wanted some querying then PostgreSQL looks great, with Amazon handling the maint and backups.
Re:Startup Engineers and Our Mistakes with MongoDB by bdcravens on Wednesday July 19, 2017 @02:26PM (#14806621)
You can even query flatfiles in s3 using SQL via AWS Athena.
Re:Startup Engineers and Our Mistakes with MongoDB by emodendroket on Wednesday July 19, 2017 @01:24PM (#14805994)
Sure, but it is also another service to keep running. Depends on what scale you're operating at.
Re:Startup Engineers and Our Mistakes with MongoDB by mrtksn on Wednesday July 19, 2017 @01:36PM (#14806135)
Any RDBMS is just a toolset to manage files on some filesystem. If you are dealing with some config files you can spare yourself the trouble and just load them from the filesystem.

NoSQL or classic RDBMS, these tools give you ways to manage your data on the filesystem(or in the memory?). You can spend time to write a code that processes files in a folder or you can use the software that does it for you. You can write a code that relates two files in a folder or you can use a software that already can do it and you can just ask that software for the outcome.

Re:Startup Engineers and Our Mistakes with MongoDB by emodendroket on Wednesday July 19, 2017 @02:36PM (#14806702)
I don't really have to write that code though; what language doesn't already have file-manipulation libraries?
Re:Startup Engineers and Our Mistakes with MongoDB by mrtksn on Wednesday July 19, 2017 @02:45PM (#14806780)
Well I guess some people wonder why people are using ready software when you can achieve the exact same thing with a text editor and a compiler.
Re:Startup Engineers and Our Mistakes with MongoDB by emodendroket on Wednesday July 19, 2017 @03:02PM (#14806913)
That's not the same thing at all; you're suggesting an approach that probably takes more code and definitely takes more setup.
Re:Startup Engineers and Our Mistakes with MongoDB by mrtksn on Wednesday July 19, 2017 @04:01PM (#14807412)
You are underestimating the complexities of file management
Re:Startup Engineers and Our Mistakes with MongoDB by emodendroket on Wednesday July 19, 2017 @07:46PM (#14808975)
Again, depends on scope and scale. If you have a handful of items to process it's not a big deal.