This school of thought that a SWE can know the entire stack, infra to app, is coming home to roost. You are the technical debt.
Learn your fundementals, and if a relational database isn't part of your foundational knowledge, learn it now or get out.
I mean, sure, he may be a "node guy", but he uses node to make web applications. A large part of that is persistence, i.e databases. For a lot of applications it's one of the most important part of your stack, if not the most important part. It's your treasure trove, it's your crown jewels. Your app is just a thing that shuffles state around in the database.
He knew a fair bit about NoSQL databases, but why did he not even consider a relational one? I'm not trying to single this specific guy out, merely asking what people think is causing the trend of engineers over-using non-rel databases for relational data.
Edit: didn't see your edit. you're spot on.
From the outside, Postgres looks hard. It isn't hard, but it looks it, because it was not designed around a five-minute demo. I use it for everything until I have a very compelling reason not to. But I have an established library of scripts and systems to make it easy, because I am both an infrastructure guy and a software developer so the idea of "I'm going to write a Vagrantfile in two minutes flat and drop in this provisioner" is something that's top-of-mind for me. (Or I just steal the Vagrantfile from another project, whatever.) But I've made that investment and it's paying off--if one hasn't made that investment, it's scary And then doing it in production? Whether it's a hosted service like RDS or doing it yourself, you have to answer a lot of capital-Q Questions along the way and that's effort that seems unrelated to actually getting your software done.
It's not just "why didn't this person think about a RDBMS?". It's that all of these easy-answer-but-not-really systems (fill in your own) are capturing the minds of developers. The quest for abstractions and ways to make ourselves better able to do our jobs is a good one. The uncritical adoption of tools that might not actually do that is not, but that's a cultural thing not limited to datastores.
- By the time you reach the limits of a Postgres setup, you will either be making so much money that porting it is a couch-cushions deal or your company is a smoking crater in a hillside.
- SPAs are fine. SPAs plus server rendering is often a pain in the rear. Walk, then run.
- If you can't express concretely why you want a container fleet, you don't want a container fleet. Your cloud provider's autoscaling will suffice. (This advice is different for non-cloud clients, but the overhead of leaving space in a container fleet for new deploys or for scaling is deadweight and turning around a container in thirty seconds is probably not as important as the blogs you read tell you it is.)
I think this applies to most relatively new (hype cycle) technologies.
Although my business partner and I have talked about a "fundamentals of cloud system design" course on Udemy or something...
Side topic: AWS did agressive move to throw it's resources to University, and pays off hansomely. Those new blood coming to our profession now has the concept of infrastructure ~= AWS (mileage varies between people, but that's the idea).
Selecting the right framework, platform, model for current needs with reasonable forward looking support takes experience and strong formal fundamentals.
On the bright side, relational is coming back with RDS, cloud-sql, spanner and cockroachdb. With any luck NoSQL will be forgotten :)
Yes, thats the way an engineer should be making decisions about arguably the most important part of the application.
(And it wasn't me that used "brogramer").
SQL as a standard is roughly followed, and PostgreSQL is very compliant. That's more than can be said about any non-relational database - each has it's own query language.
I use JSON fields and arrays quite a bit, but that doesn't mean I need a document store. It means I have small portions of data that is non-relational, or is better suited to use arrays/json than strict tables, but I still want everything else consistent, enforced, and also have all the other bells and whistles Postgres gives you (FTS, partial indexes, rich types everywhere, ACID, the list goes on and on).
Being tied to Postgres is often better than being tied to a document store. At least you can dip your toes into non-rel as little or much as you like (where it makes sense), rather than diving in at the deep end and realizing you can't swim.
> When it came to the different types of NoSQL databases, it was pretty clear that a document store best suited our needs. We needed to be able to query based on the data so key/value was out, there was no need to have the ability to make all of the complex connections between data of a graph db, and nobody actually knows what a column db is (jk but our data is not the right use case for it). So document it was. We would have to change our data layout a bit to really work well with most document databases, our current data structure was similar to what you would expect for a relational db with tables etc since dynamo also uses a table structure. I wanted something that was ACID, none of that eventual consistency nonsense. That took a lot of them out of the race right there.
I don't get it. He needed arbitrary querying, relations, ACID, tables. And yet... his first thought was to go to a document-based no-sql database and is only later hit with the sudden realization that his relational data is bet suited in a relational database.
What has gone so wrong here that an otherwise smart and educated engineer makes these kinds of fundamental mistakes? Is it the success of no-sql marketing departments, and the apparent un-sexiness of traditional relational databases? Is it education? The job market? A fear of SQL (that you never really have to write anyway...)? Is it the quick-fix non-relational databases give you?
As far as I can see there are very very very few nails that a PostgreSQL hammer cannot hit. I guess that's part of the point of the article, but why does it need an article to explain this?
How can we stop these kinds of articles from needing to be written?
> The change drastically increased the speed of our API, some requests are up to 6x faster than they were with Dynamo. The account details request, our previously slowest call, took around 12 seconds with Dynamo and was reduced to 2 seconds with Postgres!
sigh
Edit: Not sure why this has been moved to the bottom of the thread. I wasn't trying to single this particular developer out, I was merely commenting on this troubling trend. Apologies if this came across in the wrong way.