Fascinating statement which I feel is probably agreed upon by many. I can see how that can be true; but once again,I feel, demonstrates that HN has a specific subset of "developers"/"techies"/"enthusiasts".

In any "Enterprise" environment I've ever been in - core back-office enterprise resource planning apps or similar - SQL is right there front and center. "Should" it be? I don't know, I'm not a theoretician; but all developers around me for 20 years have had near-DBA level of SQL skills. Their core skills really are a) Understanding business requirements and b) Understanding SQL. For any complex business requirement, by time, we find in transactions code takes 1-10% and database activity takes 90%. So skillset in optimizing database performance of your code vastly outbenefits the skillset of optimizing your code. As much as possible, we push everything to database - queries, looping, even business logic if we can - because it's a robust, mature, optimized product and we don't have to invent optimizations from scratch...

Edit: If it may prove illuminating, even team leads & management on any of the 30+ projects I've been on, understands runstats, index reorg, transaction isolation, database maintenance basics, and a few basic optimizer dos and don'ts. They just wouldn't be able to survive without that knowledge - it's 90%+ of both good stuff we develop, and bad stuff that goes Bang in the night (e.g. a process which worked for 10 minutes for 2 years but suddenly decided to asymptotically explode and not finish before heat-death of the universe :P)

I think the phenomenon you're describing is exactly what led to the rise of no-sql dbs.

SQL allow you to do too many things, a lot of things that could happen in programming. From a design point of view, that's not a good thing, you want a data store to do just what it is - a data store, all joining and such can happen somewhere else - just make sure it's in the same transaction.

I see many of the problem stem from a undesirable design of the tables - not splitting them when you can, having too many foreign keys, and generally being a interconnected mess. Having dealt with nosqls, I attest that it is much better in that it forces you to design things in an atomic way, preventing many of the potential problem you might have down the line from the beginning.

>>From a design point of view, that's not a good thing, you want a data store to do just what it is - a data store, all joining and such can happen somewhere else - just make sure it's in the same transaction.

See, that's exactly why I indicated I'm not a theoretician and do not feel comfortable making such a sweeping "Should" statement (much as I would've made an opposite one:).

Immediately my question is "Why?" and "What data do you have to back it up?". I understand model-view-controller etc is a design pattern, but are we certain that it's always the right one?

From my trenches perspective: relational RDBMS has been around for neigh half a century. It's an INCREDIBLY mature, optimized, understood (by experts), common, standardized (for all the individual RDBMS differences), safe technology. I have team members who have been doing hard-core SQL for 20+ years, and a market full of similar, serious, hardened experts. And it'll survive changes in the higher-up stacks - in my own meager career of ~20 years, relational RDBMS has been the most stable part of the stack.

I can get a 3-5 year 'expert' on a particular programming stack, or a 20-year expert on SQL, who has seen things and will safeguard customer's data and business priorities with their life. Again, a very personal experience, but the dozens of clients I've been with, have broadly similar priorities, objectives and concerns on database level; and occasionally vastly different ones on layers above. It's a brilliant unifying common denominator.

Without a fun discussion over a drink and whiteboard, and it could be my inability to see forest for the trees, but I'm just not convinced that "all joining and such should happen somewhere else" :-\

[note, I didn't downvote your comment - I don't necessarily share the same conclusions, but I think it contributes to discussion :-]

I think your point is perfectly valid, I put should when I might have said ‘I Believe should’. But nevertheless I stand by my own point and is not quite convinced, both of us bringing anecdotes from our own experience. It is always good to hear different opinions, though.

I strongly disagree. For many backends/purposes, SQL is really the thing doing all the real work and the backend's just some fancy authentication and validation layer.

Stick close to the technology doing the heavy lifting.

If people abstract away SQL too much they just start reimplementing things in buggy underperforming code, using 1000 lines of Go or Java or whatever what should be done in 20 lines of SQL.

Also, don't get med started on people thinking they can abstract away tye database for testing. The automated test suite should always include the real database you are working against and all queries executed. I will never write a backend again without testing against the real database (whether sql or nosql).

Some people justify abstracting away the database saying "one needs to be able to switch to an arbitrary storage engine". Well..storing things is probably the main purpose of your backend. I sometimes turn that around and say, "I want to write things in SQL so that I can easily change what language the backend is written it".

Testing against the real database? Won't keeping a copy of the database just for that be expensive?

Also, should devs really be seeing potentially sensitive data?

On a side note, "I use SQL so I can change the backend language" is the hottest take I've heard in some time heh.

With database I meant an instance of whatever database engine you are using, not the actual data.

My current project uses mssql. Each test run probably spins up and destroys 100 databases (inside one mssql docker container that is spun up for each test run). Each test function populates a DB from scratch (using the same sql migration scripts that we have used in prod), runs the test, drops the database.

Can do that quite some times in the minute it takes to run the full suite.

Point is to actually execute the SQL (or whatever NoSQL you use) as part of the test.

And that's a problem - my opinion remain that individual sql queries are better left for analyst and data engineers to craft so they can discover whatever correlation they might be interested in. But for a full scale app it shouldn't be in the forefront, at least not in the same package as other logic.

The entire idea of programming is the one where you abstract away things and then abstract away the thing you're currently writing. I agree that it has gone too far in some of the places, but abstracting SQL away is the first and almost one of the best abstraction you can make.

Analysts and data engineers these days mostly work on seperate data warehouses? Do they get involved in the OLTP work in your case?

I was talking about using SQL specifically for the OLTP workload not analysis. If a query that is necessary for a backend response in some REST API ends up taking 10 minutes, but could have taken 100ms if the backend developer just knew what they were really doing over in the SQL land....the backend developer will probably waste a lot of time doing silly things with the tools available to him/her (perhaps introduce redis to cache stuff there is no need for caching, build their own fragile homegrown index table or aggregation table manually using 100 to 1000 lines of backend code plus tests, and so on).

I mean, noone ships a totally broken application that noone can use because it is so slow then asks the SQL expert to optimize it onve it has shipped!! Instead any lack of knowledge of underlying database will just mean reinvention and needless and fragile cruft in the backend code.

Note: I am against abstracting away the specific database technology you sit on (whether some SQL or not). If you are on SQL, know about indexes and materialized views (and their limitations) and so on and use them to implement efficient API endpoints, let your model in SQL directly handle any questions about idempotency or races in your API, and so on.

If you are talking about just abstracting away the SQL syntax, not the DB technology as such... such abstractions I am "meh" about, not very against them but they are usually very leaky so... They don't seem to bring anything fundamental for or against just syntax candy (getting in the way in my case).

You can't optimize well enough if your abstraction layer is too far away from your application layer.