Why is pushing the joins "to the web" inherently bad? It's a lot easier to scale web servers than database servers.
Sending that same JOIN to the web server means, at the least, shipping everything from both tables matching the criteria and matching up those two potentially huge blobs of data with each other on the web server. Doing this in an even partially efficient manner will split your query logic between DB queries and other code. Also, it's not unlikely that you'll be doing this JOIN operation in a language that maybe isn't great at optimizing that sort of CPU intense behavior on its own, so you're probably going to need to drill in and optimize on your own (vs letting SQL do its magic). Plus, we're going to need to store all this data in memory while we match it. So now our web servers, while more easily scaled than a DB, are actually pretty damn big boxes with decent memory and CPU, and that's expensive to scale.
We've got moves, and each move has multiple stops:
move: {
shipper: string,
consignee: string,
stops: [{
address: string,
appt: datetime,
}]
}
So far this looks great. I like this. Yay MongoDB!But I need to add a driver to each stop... Where do I put the driver? If I put him under the stop I'm going to duplicate his data all over the place. Do I put the move under the driver instead?
driver: {
name: string,
id: string,
moves: [{
shipper: string,
consignee: string,
stops: [{
address: string,
appt: datetime,
}]
}
Ok, not bad. We've got an array of objects under an array of objects but I'm still perfectly comfortable.Now I need to get a terminal-based look at all the drivers, so I know what every driver is doing in a particular city, also drivers can actually have multiple trucks too:
terminal: {
city: string,
drivers: [{
name: string,
id: string,
truck: [{
plate: string,
moves: [{
shipper: string,
consignee: string,
stops: [{
address: string,
appt: datetime,
}]
}]
}]
}]
}
That's getting a touch nested but... Drivers actually work for vendors, and vendors might work across multiple terminals. Do we duplicate the vendor across terminals, or the terminal across vendors? Also even the driver/truck relationship is many-to-many... Do we duplicate the truck info? What happens when we need to update a truck? Do we attempt to update across all drivers? Do we duplicate that data too? terminal: {
city: string,
vendor: [{ // many-to-many! duplicate vendor data?
id: string,
drivers: [{
name: string,
id: string,
truck: [{ // many-to-many! duplicate truck data?
plate: string,
moves: [{
shipper: string,
consignee: string,
stops: [{
address: string,
appt: datetime,
}]
}]
}]
}]
}]
}
Ok, this schema is starting to look like shit and it's only going to get worse. The shipper and consignee should actually be unique objects as well, and we might want to update the contact info on them. Do we duplicate that data and update across all moves?Also we want to cross-reference the shipper, consignee, billing, and ordering entities for market analysis (the consignee isn't always the customer, and the customer doesn't even always get the bill!) Do we duplicate that information across moves? Do we duplicate moves across entities? More many-to-many relationships...
Maybe I'm just not creative enough to use MongoDB, but I'm really starting to feel like this is a square peg in a round hole.
https://martinfowler.com/bliki/DDD_Aggregate.html
If you look at this from a Domain Driven Design standpoint, you should model your business and think about your aggregate roots and each collection should be its own aggregate root, data access should be handled by one class/microservice for each aggregate root.
If you modeled your Domain correctly that wouldn't be an issue.
On the other hand, not modeling your domain correctly first would lead to an ungodly, untestable, tangle of stored procedures and 10 way joins with A relational database.
The Mongo docs goes into best practices which mirrors basically the concept of an aggregate root.
So... It's like a join, but with two queries instead of one, and you push the join data to the webserver, which brings us back to the start of this comment thread...
Even if I were using a relational database, I would still separate out my different domains (drivers, terminals, stops, etc.) into different classes/microservices and join the different entities in memory.
This would allow different developers, teams etc. to focus on separate business objects and would allow you to start off with a monolithic code base/app and separate the app out to microservice later without intertwined logic and to mix and match storage based on what makes sense.
I never said always embed everything into one document. I said think through your aggregate roots to decide how to design your apps and your document model.
I've designed this way for the last 10 years - 9 using RDMSs. I don't do complex 10 way joins for apps. I separate out business domain logically and let each module be responsible for its own domain and then have an overarching orchestrating class that combines the various objects when needed.
That hits home. There's a lot of this in the legacy code base, and yeah, it's painful.
Interesting approach. Thanks for the discussion. I'll have to ponder this a while.
Still, having a proper transaction manager and query planner will give you far better performance compared to doing it on webservers.
Also, aren't bsically moving the complexity of the database to the webserver? Which makes your webserver far more complex and harder to scale in comparison. Especially if you need ACID. (which is a must if you are doing anything with data storage imo).
When it comes to "read" the data, you read it only from processed database that can be anything.
Since you can do your processing completely independently of your web server, your are not necessarily pushing your computational load from the DB to the web server.
OR, i can open a file stream, serialize my data to JSON, entity by entity, and dump all to a file.
The good old file.
Your position can be applied to all databases. Why not abandon them all and just use CSV ?
Personally, I never got into the hype with the NoSQL and for me the only use is as a convenient API for storing JSON files during prototyping so that I can later decide how my data should be structured.
Definitely tech debt for the future but fast iterated design that needs the database stuff fixed has more chance for success than the rigid design with the perfect data architecture.
NoSQL or classic RDBMS, these tools give you ways to manage your data on the filesystem(or in the memory?). You can spend time to write a code that processes files in a folder or you can use the software that does it for you. You can write a code that relates two files in a folder or you can use a software that already can do it and you can just ask that software for the outcome.
This mirrors my experiences with Mongo as well. The vast majority of data is relational. Mongoose allows people to make the mistake of structuring their data relationally. But all you are doing is pushing all the joins to the web server (which in Node land is single threaded and compounds the mistake of choosing NoSQL).
There may be some microservices that Mongo is great fit for, but it should not be your core data store.