Back in 2010 when the MongoDB hype was high, the well-known company where I was working at the time decided to build the next version of the product using MongoDB. I was on the analytics team and had to code a whole bunch of intricate map-reduce jobs to extract summary data out of Mongo. I'd repeatedly head to the product team and ask them to explain the edge cases I was seeing in the data and they would not be able to give me an answer because the data was on the third or fourth version of their mentally-stored schema and no one knew anymore. All in all, misery.
Iirc, they've turned around on their anti relational views and now allow for joins.
I looked at OrientDB awhile back ago, but it fell flat with the lack of features and oddness.
If I had more time, I would really dig into ArangoDB.
Since they are both of the NoSQL family.
People were convinced that Mongo was a good choice as a default, general purpose db, when it clearly wasn’t for about a million reasons.
I don’t think DynamoDB is marketed or viewed in the same way. The docs are pretty clear about needing to design your data model to specifically work well with Dynamo. People using it seem to generally be aware of its limitations, and deliberately choose to accept them for the sake of performance and scalability. At least that’s my perception.
More importantly, AWS is very explicit in letting newcomers be perfectly aware that DynamoDB's usecases are, that they are very specific niche use cases, and that if users require schemas and joins then they should just stick with either relational databases or graph databases.
Those who overpromise will always disappoint.
Web scraping or data ingestion from apis might be a reasonable use case. Or maybe consumer apps/games where occasional data loss or inconsistency isn’t a big deal.
It can also be used effectively as a kind of durable cache (with a nice query language) in place of redis/memcached if you give it plenty of ram. While its guarantees or lack thereof aren’t great for a database, they’re pretty good for a cache.
I went on to work at mongodb
It's cheap, its fast and it scales super high. Don't need much more
DDB is great for storing data that is required to be scalable and never needs to be joined. Add in DAX, developer time necessary to orchestrate transactions, calculate the scaling costs and...that's how AWS gets you.
Plus, local development requires half-complete emulators or a hosted database you're charged for.
No, maybe people should think twice about DynanoDB.
So far so good. Being able to join on JSONB fields right in the SQL is awesome
At least with the ORM(-ish) tools I worked with, it always felt much more straightforward to just change classes within the application code and automatically generate the respective migrations files to be run on the relational database, than having to interact as a human with json (for the app as well as for business intelligence and reporting/monitoring).
I feel I have handwritten significantly more schema and data migration code for non-sql databases and JSONB in Postgres than for relational databases in the last 10 years.
Sure once the database growths bigger, those automatically generate migrations files don't work as seamlessly anymore and can be dangerous, but no tool or database magically solves all the problems at scale.
There's your answer, no? You've got specific tools and workflows you've designed to work with your database in a specific way. You can't just take one workflow and substitute a piece of another workflow and expect it to always work. Your tools and processes would be as useless for me as mine would be for you.
As in you need to switch your entire database system or make a bet on this completely new player instead of battle proven tech, because with relational databases schema migrations are hard. Among a bunch of other questionable claims.
Sure if you take the vanilla databases without their ecosystem, there is some merit to that, but I don't find it a very practical argument to ignore all the tooling and workflows that do exists and are used by people.
When MongoDB first came out, I was eager to check it out. I was using mainly Django with Postgres at the time, but had my career start on ZODB, an object-oriented non-sql database created in the late 90s. So I was hopeful that MongoDB could give me the best of both worlds. One of the reasons I backed of Mongo very quickly was the lack of tooling as soon as you have something resembling a schema or relationship and you'd want to change it.
The appeal is non-technical.
Writing good migrations, having tests around them to ensure they didn't leave the DB in an inconsistent state if a subset of them failed requires good understanding of a RDBMS and the specific product.
You'll be surprised how many engineers don't meet that criteria.
The JSON to SQL columns workflow allows any developer to offload what a RDBMS does into the client/server/bootstrap code.
At that point the DB is like a key-value store and you get to claim you are using Postgresql.
Is that what I would use if I had reliability and performance in mind?
No... but it's way cheaper (and quicker) to find devs who can do this (JSON to SQL columns workflow) than devs who can write well tested, reliable and non hacky migrations.
Often at a startup, reliability and performance have lower priority than getting some features out the door in a day.
I know this statement is a little tongue in cheek but there is a lot of value organizationally to using Postgres as a key-value store vs MongoDb or some other flavor of no-sql. Even if you're not really getting all of the benefits of a relational db, you are still getting ACID transactions, envryption, user/access controls, backup/restore, libraries, etc. You're getting this in a tool that you are comfortable with and that you may be using in other areas where a relational model is more appropriate and thus have experience with a lot of the management aspects. Obviously if you are a mongo expert and using mongo everywhere this doesn't apply but in my experience software engineers/data engineers/DBA with postgres experience are easier to find.
> in my experience software engineers/data engineers/DBA with postgres experience are easier to find.
I need to understand what "postgres experience" means in this context. Having experience writing postgres client software using some postgres driver is a different kind of experience than installing, provisioning, monitoring and turning postgres itself.
The former is a days worth of work for any good engineer; the later can take decades and those people are really hard to find.
> you are still getting ACID transactions, envryption, user/access controls, backup/restore, libraries, etc
Sure, at a high level, making a KV-store on top of a RDBMS feels like a grand idea - hey you get ACID for free and all the RDBMS goodness while getting the flexibility of a KV-store to boot.
As they say - when something sounds too good to be true ...
One small scenario worth thinking about:
What does a KV-store on top of a RDBMS with ACID semantics even mean?
How do you handle merging objects?
Does the RDBMS even supporting merging objects inside a column?
What would happen if I have the different attributes in the same object that I then update one after the other into a JSON column on a RDBMS?
Does it fail one of the updates with an error message that data is lost?
Does it automagically merge the data? How does it even know how to handle conflicts?
Hint: No it cannot - the reason why a RDBMS can even offer ACID semantics is because RDBMS operations assume certain rules have been followed. Database Decomposition is important for a number of reliability reasons.
There are KV-stores out there that handle all these scenarios natively without the client having to worry about it (and doing a halfassed, buggy job about it because they are not DB designers). They have conflict resolution algorithms, offer CRDTs for the clients to use and engineers who know what they are doing, use them.
Using the right tool for the right job is what experienced engineers do because it not only makes their lives easier but the businesses they help run more reliable and resilient.
There is no magic.
I'm using it because the cost of changing the schema for a feature that may only be experimental is too high. I started off changing the schema every time, and it just got too expensive.
Using the JSON-to-SQL workflow, I can mess around with feature design and iterate fast, and then crystallise the schema into SQL once it has stopped changing.
Writing solid migrations can end up being more complex than the feature code itself, and when features are changing fast, writing solid migrations is kinda pointless. The feature may not even make it to production.
But yeah, sure, I'm an idiot who doesn't know how to write proper SQL. rolleyes
Everything or certain parts of it?
> Using the JSON-to-SQL workflow, I can mess around with feature design and iterate fast, and then crystallise the schema into SQL once it has stopped changing.
Sure, you're experimenting and things are in a state of flux.
I get it. Been there done that, but not in production.
Your customers want features over reliability. This is perfectly OK at a startup - the customers already depend on a system elsewhere but that system does not meet all their needs so they are experimenting with you but if your company runs to the ground tomorrow, they still have not migrated 100% over to you, so they are "safe". If your software loses data, they probably won't even notice until they really start to move over for real and lose money (if they did not do testing for consistency).
In your position I would be honest and say data can be lost or corrupted and thats the price I am willing to pay for flexibility given the resource constraints I have.
It's too expensive to do things properly and customers don't really want that reliability so we can take some risks.
However, it would be naive to say you just designed an ACID complaint KV store by implementing some adhoc KV operations using JSON columns in a RDBMS.
I make a lot of money based off the disasters software like this creates when the original architect of such a system have "moved on" to other companies.
Data corruptions, losses, inconsistencies, meaningless key relationships, subpar performance (one place had two columns - an ID and a JSONB and were wondering why their query performance was poor, the DB locked so much and some updates never appeared to go through or "reset" the data).
There are KV-stores out there, that sell for a pretty penny, handle all KV conflict scenarios natively without the KV-store client having to worry about it (and doing a halfassed, buggy job about it because they are not DB designers and might not even know the bombs they are planting in their code. Maybe ignorance is bliss?). These systems have conflict resolution algorithms, offer CRDTs for the clients to use and engineers who know what they are doing, use them.
> The feature may not even make it to production
Feel free to do whatever you want with code that never makes it to production and affect people's livelihood.
I would argue you don't even need a DB to give you the warm fuzzy feelings and just do everything in memory.
Afterall, memory is cheap and configuring a DB correctly can get too expensive.
As an aside - if you're working at a company that regularly pushes features that do not even make it to production, there's a miscommunication issue.
Your business is bleeding money.
This is not to say overall, your business is not profitable - its just that it's bleeding money in that specific project and other more profitable ones are making up the slack.
At the minimum, the deliverable should be broken down into a POC and, when it's clear a production need absolutely exists, that POC is delivered production ready.
> Writing solid migrations can end up being more complex than the feature code itself
Sure, thats the price you pay for ACID compliance.
Nothing is free. There is no magic.
> I'm using it because the cost of changing the schema for a feature that may only be experimental is too high. I started off changing the schema every time, and it just got too expensive.
Again, you are free to do whatever you want in an experimental setup but we are talking production here.
If your proposal is you push the same design to production that you use in your experimental/POC setup, in a serious, well used and depended on production environment without anyone noticing, I would really like to know more! My contact info in the profile.
If your claim is you have figured out a way to get something for nothing - hmm.
The ROI of the businesses you help run just trended towards infinity!
I know what I'm doing. I've been doing this for literally decades. Please stop assuming that people who don't agree with you are doing so from ignorance.
I listened to what you had to say the first time (which is why I responded to it) and will continue to listen to what you have to say. My contact info is on my profile.
> Please stop assuming that people who don't agree with you are doing so from ignorance
I make no such assumptions.
I shared why I think the JSON blob on a RDBMS causes more pain than it solves and I look forward to your refutation of everything I wrote.
My contact info is on my profile and I keep it there because I like to know where I am wrong.
Have a great weekend and look forward to connecting with you
I change the application code, trigger the cli to generate the sql migration and be done with it. In 99% of cases I don't even look at the generate file, I don't write specific migration tests, etc. If migration runs without error and the application tests pass in development and staging there is a good chance it will as well in production or auto rollback saving my ass.
This works until it doesn't and you have to become much more vigilant. But at that point I much rather build upon the relative robustness and consistency of the sql I got so far, than JSON that is all over the place.
Maybe it's my tooling, but almost every time I made a JSON column in Postgres for the sake of saving time in the past, it slowed me down almost immediately after adding it.
Yup.
> almost every time I made a JSON column in Postgres for the sake of saving time in the past, it slowed me down almost immediately after adding it
Backs my experience fixing software from companies that had terrible data loss issues after these JSON hacks reared their ugly head.
I am not saying JSON does not belong in a DB - I have many well tested and audited code that does do that. What I am saying is I have seen things go wrong far too much, for me to trust that someone who proposes that approach to actually know what they are getting themselves into.
Plus, it's way cheaper to use a custom built KeyValDB that actually supports object semantics if one does not mean to use RDBMS semantics.
Anyways, I am being severely downvoted and cant respond until a few hours between posts so I wont be responding any longer on this thread as I have a lot of work to do the next few days and cant wait around.
See my other comments. Contact info in my profile, would love to be in touch!
I've done a couple projects where we kicked off with postgres using JSONB columns for early iteration. Then we gradually migrated to normal SQL columns as the product matured and our design decisions crystallized. That gave us basically all the benefits of mongodb but with a very smooth journey toward classical database semantics as we locked down features and scaled.