It starts really early on. When I first used MongoDB I thought to myself "what a fresh breath, I can finally ignore normalization!" The documents I inserted would contain all the information I needed. One retrieval and I got what I needed. Wow!! But then I started doing sorting, searching and I had to do most of the work on the client side (my backend). At that point, I found myself in trouble because in my other tables I also dup my information. Data has to be updated in multiple places. So I thought "let's take out the dup data." Then I found myself not knowing how to structure my document data anymore... back to some sort of normalization. At the time the searching and ranking in MongoDB were also poor, so I was forced to doing the entire thing on the client side regardless.

I went back to PostgreSQL since. I probably needed a good one-on-one expert training with MongoDB, but I just found myself happier with RDBMS. You don't have to be strict normalization in RDBMS, just enough to make sense for your use cases.

One thing I really did like about MongoDB back then was storing blob (files). It was the best solution available without setting up S3. At the time there was a limitation with 4GB (??) but MongoDB worked for my use case anyway. That being said, please don't store files in any databases today. Use DB to store references to a real object storage like S3. When the DB crashed, you better hope no corruption.

It amazes me how quickly our industry has forgotten the need for DBAs. With these MongoDBs, MPP cloud dbs and Hadoops, everyone seems to have assumed that engineers can now do all db work. This is reflected in the titles too: Data "Engineer".

But from my perspective, this is delusional. There is a lot that goes into DBA's experience that is not solved by the performance improvements in databases over the past decade. But there are more choices. 20-30 years ago, you would have been forced to write code on Oracle and you would have asked for help before deciding how to structure the data. Today, with more choices, you just read some online opinions, and jump on it without any internal resource to guide you.

Not saying the world of Oracle was great, but the young on this thread (me included) would benefit from respecting the experience of the old.

I agree. The issue I see is it's pretty tough to hire very good DBA that knows the new cool technologies to be very honest. I've worked with some DBAs but they have no experience with Cassandra or whatever !MySQL !Postgres !Oracle !SQL, and they also have very difficult time integrating themselves with the developers. It turns out the developers have better understanding of Cassandra than the DBAs and DevOps/Ops. As Ops we just learn from them and from incidents.... Good DBAs also tend to do a lot of testing and development besides reading manual and utilizing past experiences.

> everyone seems to have assumed that engineers can now do all db work

Yes, everyone wants "full stack" and so you have a bunch of people haphazardly adorning themselves with the "full stack" label.

It's a gradual process and you start feeling the pain fairly early on.