I could say I'm surprised to see how many people don't know anything about the browser rendering pipeline and consequently produce badly performing interfaces. Someone else would complain about people's inability to do even the most basic Linux administration tasks.
But at the end of the day any single bit of knowledge doesn't say much about your general ability to be a developer. The term "web development" covers an extremely wide array of topics. Luckily, there are many wonderful pieces of software out there that abstract things away and allow you to produce a finished product even if you aren't an expert in everything involved.
Making either of those your line in the sand for what a developer should and shouldn’t know is arbitrary. I don’t know a ‘Scala’ full-stack developer that could do advanced optimisation on React render times, or a ‘React’ developer that knows transaction isolation.
Not knowing either of these does not make you an unworthy developer and unless you have a specific need for these skills it probably does more harm than good expecting everyone to know them.
It's very expensive to acquire customers. It gets even more expensive when you have dirty data from incomplete domain modeling.
At the most basic conceptual level, the business of software rests on reading inputs correctly, processing them, and writing the right outputs with strong guarantees. SQL is historically the most sophisticated, powerful, common and cross platform way to do that in a structured manner that is easy to reason about for a vast majority of use cases. You can layer an application on top of the foundation of an appropriate data model, but the appropriate data model is a root requirement to even get to a passable prototype. That is why gradual mastery of it yields such outsize payoffs compared to mastering other skills.
What about security? Your business won't succeed if you're hacked daily.
What about monitoring? If you're losing data and don't even know it, you're creating compound technical and potential business debt.
What about performance? Once you have a perfect data model, no amount of trying to optimize it will improve your business further. Optimizing other parts of the application can.
Basically everything is important, saying that just because someone isn't an expert in one phase that they're not as good of a developer is myopic.
The truth is, you need to be good enough at all of the high priority parts of technical competency. Security, monitoring, performance -- sure, all of those are also extremely important. They're all places where unforced errors can be introduced that can and do hurt the business -- sometimes catastrophically or fatally.
With that said, to not recognize the evergreen utility of domain design skills is ignorant. You will never get the luxury to worry about security, monitoring, or performance if you don't build the state machine that makes the right outputs out of the right inputs because you will either never sell it to a customer, or lose that customer when they choose a competing solution which actually does what it's supposed to do.
It's not true that everything is equally important. I think that's a myopic way to look at software development without considering the business impact of key crucial areas where software design and maintenance intersect with stakeholders.
Which can be the consequence of setting the wrong isolation level: http://www.bailis.org/papers/acidrain-sigmod2017.pdf
The distinction is similar to a property developer and a civil engineer. Both create buildings, but one does it at scale by offloading functions to known entities and prepackaged solutions, while the other understands one domain in depth.
Both are needed in any team or organization, because not every solution needs to be "engineered" (a Dockerized Redis instance without SSL or auth behind a corporate firewall may survive untouched for a decade), but sometimes you have to engineer something that withstands gale-force winds at 1000 ft height.
I mean, this is still arbitrary. There are millions of small businesses worldwide that do fine with terrible records, but making on-going sales.
If you don't make any sales because of constant outages, you could argue that's worse than trying to deal with corrupted data, especially if you have log files or some other method to recover corrupted data anyways.
My point is that the very root of what gives software value is something deeply industrial -- it is a machine that can do something over and over correctly, reliably and without regressions. If you take away that reliability to be depended upon by a business, you are no longer building an industrial strength solution -- you're building a toy.
As long as you have both covered to a reasonable degree within your company/team it’s a waste of time arguing who’s got the most important info in their brains.
SQL is intrinsically very close to the metal of your business logic, especially when it comes to general business logic. General business logic is typically most critical and risky around the surface of payments. A JavaScript bug on your site around your shop preventing checkouts is very bad because you are indeed losing cash by the minutes. A poor domain model that doesn't properly associate active promotions to cart order lines could result in an incorrect charge amount and an irate customer.
if you can recommend a good source for learning the browser rendering pipeline please post it here, everyone I have asked just shrugs it off and says that no one really knows.
Other than that there are some older docs for the Blink engine which drives Chromium. It's much more low level and hard to follow though[1][2].
The Navigation Timing spec is good for building an understanding of the major events that go into a page loading and creating HTML elements[3]. It's not the whole picture but gives the timings for navigation and DOM element events.
[0] https://www.udacity.com/course/website-performance-optimizat...
[1] https://www.chromium.org/developers/the-rendering-critical-p...
[2] https://docs.google.com/document/d/1wYNK2q_8vQuhVSWyUHZMVPGE...
[3] https://www.w3.org/TR/navigation-timing/#processing-model
https://developers.google.com/web/fundamentals/performance/r... https://developers.google.com/web/fundamentals/performance/c...
Would be interested to hear of more also.
https://www.html5rocks.com/en/tutorials/internals/howbrowser...
https://developers.google.com/web/updates/2018/09/inside-bro...
I mean, if you're supposed to be a senior front-end dev then that wouldn't be an unreasonable expectation. Meanwhile there's plenty of senior back-end developers out there working on SQL-backed applications who don't know SQL and refuse to learn it.
No one cares about that because the website will be redone in the hottest new framework every few months anyway. Whereas serious databases are around for decades.
It’s a surprisingly good filter.
EDIT: To clarify since this got more attention than I expected...I don’t disqualify any candidate based on the answer to a single question. I just use that question to assess their actual SQL experience. If you’ve spent any amount time writing raw queries by hand for reports or just to pass through to a web service, you’re going to have run into HAVING. It’s a simple part of the basic SELECT syntax without needing to involve joins, subqueries, index optimizations or specific knowledge of database internals.
It’s a fair question and the only way you wouldn’t have run into it is from working purely with ORMs or NoSQL. There’s a lot you can do without it, but it goes a long way in determining my ability to ask you to open up a query analyzer in different environments. It’s also going to tell me a lot about the way you think about data problems and where you are going to gravitate for certain types of logic.
I just don't use it enough to keep it in memory as a first class concept. A quick google search with an example is enough to jog my memory.
If I got asked about it on an interview, I'd be completely blank.
"I use [x] all the time and know it well, so [x] will be a good filter for evaluating someone's technical skills."
No. People do different work tasks depending on the project or company.
Sometimes I'm knee-deep in SQL. Others in React. I'm often forced to do PHP. My familiarity with specific SQL syntax ebbs & flows with what I'm doing.
There's no point in keeping something in your brain if you're not going to use it every day. Just remember the general concept and Google the specifics when you need it.
Just one example why interviewing is terrible and interviewers really should be trained on what not to do.
Need me to write an ETL. I probably can't crank it out in an hour like a data scientists, but I can get it done in a reasonable amount of time. I just need to do some major context shifting.
Have me write only ETL's for a week, I'll be cranking them out after the first few.
WHERE filters the data before you aggregate it, HAVING afterwards.
That's all there is to it.
It’s like asking if intel x86 is big-endian or little-endian. It’s trivial to check but if I haven’t had to use that fact in a while, I likely won’t bother to remember because it’s trivial to search for.
That does not mean that I do not know basic concepts about how different databases work, including underlying storage engines etc. To the point that for one product I personally designed actual NoSQL (EAV to be precise) database engine and query engine largely resembling SQL to go along.
So would you fail me in your test?
I can describe for you in detail the different index types on MySQL and Postgres, and go into when/why you would use each. But it's been so long since I crafted a complex join by hand I would definitely have to google it. However I don't doubt that I could write the query in a short time. One of the most realistic interviews I ever had, they gave me a laptop and allowed me to google during my answer. I took the job.
Like in Python if you asked lots of questions with list of numbers ... and then generalized it to infinite lists of numbers. If the first case I'd like to see list comprehensions, and then generator expressions or possibly the use of itertools.
There are a lot of ways to answer it, but HAVING is the simplest.
Usually when you add data to the mix, you'll find that most questions are not significantly correlated with candidate success/failure once you condition on "ability to write any code at all".
A couple years later I found the stack of resumes, some we hired, some we didn't. I realized that some of our best engineers had failed my question, and I had subsequently voted "no" (I was outvoted, fortunately, by my fellow panelists. Another reason I strongly advocate panel interviews but that's a separate discussion).
I soon realized that my question was good at filtering out people who didn't think like me, not at filtering out people who would make good engineers.
My takeaway was that everyone, especially me, could use a healthy dose of humility and self-skepticism. Not advocating swinging the pendulum into Impostor Syndrome (which I now struggle with sometimes), but somewhere in the middle is good and healthy IMHO.
I HATE trivia questions and I’m convinced they are a sign of a lazy interviewer. If you want to test someone’s knowledge of sql, give them a sample schema, an internet connected computer and five minutes to write an appropriate sql query for your use case.
I’ve never met somebody who has spent any amount of time writing raw SQL who didn’t know it.
I’ve met a lot who overly rely on their ORM but “have done a lot of SQL” who don’t.
(+) average= meaning the programmer should be able to say 'solve' the FizzBuzz under comfortable conditions. Or say add the first n items in an integer array etc...
My interviews are conversations to get to know you, your background, your professional interests, how you think about problems and how closely your resume lines up with those conversations. I like to get people talking about their work to see where their energy level goes.
When we start talking through a hypothetical data problem there are people who will describe the problem from the UX perspective, the app code perspective and the database perspective. The question prompts that portion of the conversation.
Precisely. And why not? I have my fellow developers do that ( ask trivia and label the interviewee incompetent, if he cannot answer it) , and most people who have reacted to your post have also probably seen that.
Anyway humor me and tell me how much time it would take?
Using the concepts in how you naturally think about problems?
That will only come from experience. I can't say exactly how much but I'd imagine something in the realm of 6 months minimum of applied usage to different problems.
And I interjected during a long pause and said "It sounds like you're trying to ask a question which could be answered by using a HAVING clause. So something like, only those Customers who have two or more orders, which would be 'GROUP BY Customers HAVING count(Orders) > 2'."
I was right, he was looking for that. Still didn't get the job.
It shows what happens when you have exactly the misunderstandings being discussed here. Also note that the author still has absolutely no idea how to do a simple ER diagaram.
> Once we figured out that we had accidentally chosen a cache for our database, what did we do about it?
I'm going to use that one to win imaginary arguments in my head with my pet NoSQL strawman.
I have written about both these here:
https://henrikwarne.com/2011/12/18/introduction-to-databases...
https://henrikwarne.com/2019/07/27/book-review-designing-dat...
I've used one. It was great. They (mostly) failed because of business and politics and a whole lot of other reasons unrelated to the technology. I'm sure by now everybody here knows that systems don't succeed or fail based purely on the technical qualities of their design and implementation.
SQL queries can be done in code in a sane way to avoid sql injection or you could implement stored procedures and functions to make operations more ORM-like but not pay any of the ORM tax.
It’s just annoying to see the DB treated so flippantly by devs who are Hussein Bolt quick in grabbing an ORM when it’s not always the right tool or even necessary. They’re great for prototyping but not for production imo
Also, his name is Usain Bolt[2].
The one huge win for ORMs in my mind is the standing up of and the iteration of schemas, defining a schema and then summoning a database in alembic vernacular is really neat whereas one would have to manually manage a set of init scripts and migrations oneself.
In the end there's no really clean way to do DB maintenance and work it's just work and has to be done, imo.
I did my time. I understand normalization. I've got a vague understanding of the various normal forms (except 6NF which nobody seems to understand ;-)
I can use a query profiler and I've occasionally read up on the different index types and their performance profiles.
But I still prefer to use an ORM. SQL syntax just never fitted my brain.
And the ORM I use can handle aggregation, annotation and a bunch of stuff that you couldn't really describe as "trivial queries"
I don't see how adding logic to a service at runtime is any different than fiddling with queries and optimizing them
I don’t see the point of reinventing wheels unless you have to, unless you have lots of time and energy to waste
We can easily enumerate dozens of other technologies that developers hard push on only to years later realize they’ve swung too far.
For the majority of webapps, I'm not sure the default isolation level of MySQL and Postgres is an appropriate tradeoff, since I'd rather have correctness by default than performance by default.
At that level of depth you just hand it over to the CSS champ or the DB champ.
Developers are blaming other developers for not knowing everything they do because businesses forced them to learn everything (Fullstack), but that's actually a bad arrangement and somehow the solution is to just punish the developers more rather than hiring DBAs.
> Developers are blaming other developers for not knowing everything they do because businesses forced them to learn everything (Fullstack), but that's actually a bad arrangement and somehow the solution is to just punish the developers more rather than hiring DBAs.
If the organization views developers as cogs or something like a software assembly line, the desire for interchangeable, unspecialized workers seems like a natural consequence.
It isn't purely a matter of number of users. You can have a billion users, but if your access pattern is embarrassingly isolated you are still probably okay. (And sharding is going to work great to boot.)
Any quality database abstraction provides fairly simple means to perform transactions.
And most people with knowledge of DBs should run, not walk, away from a team that operates like that...
Does the app handle user accounts? Payments? Data editing? Multiple logins into the same account? Then it should handle races and transaction properly...
That said, it's true that they can be rare, and might be less worth it (engineering or time wise) than just letting it go.
For an app with like a few thousands users or internal use, the odds might be small.
But a fairly standard web app might serve 1M, or 10M or 100s of millions of people, and they deserve better.
So less features and slower performance to account for the hypothetical case where someone tries to edit their user account from 50 devices at once over and over again?
> But a fairly standard web app might serve 1M, or 10M or 100s of millions of people
You are talking about less than 0.5% of revenue generating SaaS apps.
/s
Otherwise, you end up in the situation where every second person has a different set of pet skills and you can't hire anyone because they need to know all the skills perfectly.
But it depends on what you're doing and your market. I've worked with people who took a pay cuts to go work for FAANG, after several interviews which they've studied for but I've decided it's not really needed when you're just writing a little in house CRUD webapp.
When combined with tools that create a schema from code (which are in of themselves quite leaky), the resulting schema is frequently atrocious and lacking in both safety and speed.
For tools to generate a schema from code, the common data types are simple since the tooling is usually good enough to handle those common cases. But once you get into the non-trivial types (notably numerical types with custom scale and precision and timestamps), things go haywire very quickly. Another example is that all of the migration tools I've ever used have required the developer to define the model separately from the actual migration code which inevitably causes drift in the schema/migration code at some point.
Anyone that tries to define non-trivial relationships between tables/entities using non-SQL code is usually doomed to strange relationships in the RDBMS that'll make it arduous/near-impossible to ever use a different migration tool/ORM. I've seen some horrifying schema designs due to a near religious dependency on ORMs/automatic migration tools (foreign key references to the same table, join tables lacking foreign key constraints, columns intended to be foreign keys lacking a foreign key constraint, tables lacking in primary keys, etc.).
[0]: Notice how constraints are deferrable and adding comments onto columns are only available on certain RDBMS. https://sequelize.org/v5/manual/models-definition.html
To the point where I'm working today to fix slowness in a client's app where it relies way too heavily on the ORM and plain ruby code instead of going more in depth with sql.
Knowing sql more in further depth is the number one skill I'd want to tell backend devs to learn.
Also, I'm mainly a python guy, and if we're talking about ORMs, I will say that SQLAlchemy doesn't live up to ActiveRecord, to the point where I built my own db connection code for queries in some of the other flask / general python apps I'm working on.
Amazon’s DynamoDB for instance supports transactions and optionally consistent reads. With Mongo you can set your WriteConcern to force strong consistency. Mongo also supports transactions.
More a property of concurrency than of SQL, I believe; your point applies for programming languages too.
It's possible to get by as a programmer with no real understanding of how concurrency works in the relevant languages.
If concurrency bugs were always obvious, doubtless things would be different.
I am one of those who don't care about SQL, I assume it will fail and in general avoid transaction as much as possible.
In any "Enterprise" environment I've ever been in - core back-office enterprise resource planning apps or similar - SQL is right there front and center. "Should" it be? I don't know, I'm not a theoretician; but all developers around me for 20 years have had near-DBA level of SQL skills. Their core skills really are a) Understanding business requirements and b) Understanding SQL. For any complex business requirement, by time, we find in transactions code takes 1-10% and database activity takes 90%. So skillset in optimizing database performance of your code vastly outbenefits the skillset of optimizing your code. As much as possible, we push everything to database - queries, looping, even business logic if we can - because it's a robust, mature, optimized product and we don't have to invent optimizations from scratch...
Edit: If it may prove illuminating, even team leads & management on any of the 30+ projects I've been on, understands runstats, index reorg, transaction isolation, database maintenance basics, and a few basic optimizer dos and don'ts. They just wouldn't be able to survive without that knowledge - it's 90%+ of both good stuff we develop, and bad stuff that goes Bang in the night (e.g. a process which worked for 10 minutes for 2 years but suddenly decided to asymptotically explode and not finish before heat-death of the universe :P)
SQL allow you to do too many things, a lot of things that could happen in programming. From a design point of view, that's not a good thing, you want a data store to do just what it is - a data store, all joining and such can happen somewhere else - just make sure it's in the same transaction.
I see many of the problem stem from a undesirable design of the tables - not splitting them when you can, having too many foreign keys, and generally being a interconnected mess. Having dealt with nosqls, I attest that it is much better in that it forces you to design things in an atomic way, preventing many of the potential problem you might have down the line from the beginning.
See, that's exactly why I indicated I'm not a theoretician and do not feel comfortable making such a sweeping "Should" statement (much as I would've made an opposite one:).
Immediately my question is "Why?" and "What data do you have to back it up?". I understand model-view-controller etc is a design pattern, but are we certain that it's always the right one?
From my trenches perspective: relational RDBMS has been around for neigh half a century. It's an INCREDIBLY mature, optimized, understood (by experts), common, standardized (for all the individual RDBMS differences), safe technology. I have team members who have been doing hard-core SQL for 20+ years, and a market full of similar, serious, hardened experts. And it'll survive changes in the higher-up stacks - in my own meager career of ~20 years, relational RDBMS has been the most stable part of the stack.
I can get a 3-5 year 'expert' on a particular programming stack, or a 20-year expert on SQL, who has seen things and will safeguard customer's data and business priorities with their life. Again, a very personal experience, but the dozens of clients I've been with, have broadly similar priorities, objectives and concerns on database level; and occasionally vastly different ones on layers above. It's a brilliant unifying common denominator.
Without a fun discussion over a drink and whiteboard, and it could be my inability to see forest for the trees, but I'm just not convinced that "all joining and such should happen somewhere else" :-\
[note, I didn't downvote your comment - I don't necessarily share the same conclusions, but I think it contributes to discussion :-]
Stick close to the technology doing the heavy lifting.
If people abstract away SQL too much they just start reimplementing things in buggy underperforming code, using 1000 lines of Go or Java or whatever what should be done in 20 lines of SQL.
Also, don't get med started on people thinking they can abstract away tye database for testing. The automated test suite should always include the real database you are working against and all queries executed. I will never write a backend again without testing against the real database (whether sql or nosql).
Some people justify abstracting away the database saying "one needs to be able to switch to an arbitrary storage engine". Well..storing things is probably the main purpose of your backend. I sometimes turn that around and say, "I want to write things in SQL so that I can easily change what language the backend is written it".
Also, should devs really be seeing potentially sensitive data?
On a side note, "I use SQL so I can change the backend language" is the hottest take I've heard in some time heh.
My current project uses mssql. Each test run probably spins up and destroys 100 databases (inside one mssql docker container that is spun up for each test run). Each test function populates a DB from scratch (using the same sql migration scripts that we have used in prod), runs the test, drops the database.
Can do that quite some times in the minute it takes to run the full suite.
Point is to actually execute the SQL (or whatever NoSQL you use) as part of the test.
The entire idea of programming is the one where you abstract away things and then abstract away the thing you're currently writing. I agree that it has gone too far in some of the places, but abstracting SQL away is the first and almost one of the best abstraction you can make.
I was talking about using SQL specifically for the OLTP workload not analysis. If a query that is necessary for a backend response in some REST API ends up taking 10 minutes, but could have taken 100ms if the backend developer just knew what they were really doing over in the SQL land....the backend developer will probably waste a lot of time doing silly things with the tools available to him/her (perhaps introduce redis to cache stuff there is no need for caching, build their own fragile homegrown index table or aggregation table manually using 100 to 1000 lines of backend code plus tests, and so on).
I mean, noone ships a totally broken application that noone can use because it is so slow then asks the SQL expert to optimize it onve it has shipped!! Instead any lack of knowledge of underlying database will just mean reinvention and needless and fragile cruft in the backend code.
Note: I am against abstracting away the specific database technology you sit on (whether some SQL or not). If you are on SQL, know about indexes and materialized views (and their limitations) and so on and use them to implement efficient API endpoints, let your model in SQL directly handle any questions about idempotency or races in your API, and so on.
If you are talking about just abstracting away the SQL syntax, not the DB technology as such... such abstractions I am "meh" about, not very against them but they are usually very leaky so... They don't seem to bring anything fundamental for or against just syntax candy (getting in the way in my case).
To all the naysayers and downvoters: There's a difference between knowing SQL and being an SQL ninja or a master of database architecture. Those are three different things. The former, and what my original comment concerned, was just learning the language itself.
I concede that learning to be able to craft complicated, performant queries and design complex databases with any desired property you might dream of is not quick or easy, but I still maintain that learning the language itself (sans extensions) is.
I would not be able to do this super quickly: https://stackoverflow.com/a/42939280
If you're already technically proficient in general programming (or Excel), you're likely to pick up SQL quickly.
There is immense, unwarranted filtering of people from jobs for relatively minor things.
Realizing this allowed me to understand why:
1. There is so many used ORM(ish) libraries which make it in practice harder to access the database for anything but trivial queries.
2. NoSqlish databases seem to have became so successful even in application where non of there "benefits" (wrt. scalability and similar) matter and you would normally prefer to avoid some of there drawbacks (e.g. eventual consistency, no "system wide-ish" transaction, enforcement of correct schemata and some parts of data consistency). (Sure there are other reasons for there success, too. Like being fancy, modern or no clear requirements analysis and therefore no idea about scaleability requirements ...).
I mean if SQL and relational databases are for you just structs with a bit more basic types then JSON but which in turn are flat requiring annoying foreign key references and a bunch of ceremony around this with very little added benefits then yes it makes so much more sense to just use a nosql database and be done with.
(PS: Yes I'm aware that for certain use-cases the resulting databases can be very complex and hard to use, what I mean is that it's a skill you have to learn to use efficiently and a scarily large amount of people a came in contact with in the recent years not only don't have that skill but are not aware that, if they don't want to mess up larger databases they work on, they will have to learn that skill.)