It has always been like this. The distinction used to be: base software developer vs applications developer.
Base software developers went to work on operating systems, databases, web servers, etc.
Application developers worked on business software.
And most decent business software development shops kept a couple of devs with decent knowledge of base development just to go for when the needs got technical.
These are disparate skillsets and both of them are difficult to master. Claiming one is harder than the other is really quite shallow. Some people are going to naturally have an easier time picking up one skillset while other will have a naturally easier time picking up the other skillset. I think rather than trying to pretend like people should stop focusing on hard skills and start focusing on soft skills, we could realise for a moment that it does nobody any good to take people who are good at one task and insist that they must retrain for another when there are already plenty of people who are good at the other task.
It would make more sense to get these groups of people to learn just enough of the opposite person's skillset so that they can effectively communicate and as a result form a well functioning two-sided system where one group of people talks to users and analyses their requirements and the other group of people actually works off of these more concrete better defined requirements to actually implement the software.
If you think that "soft skills" are harder than "hard skills" it's probably because they're harder for you, that doesn't mean they're harder for everyone, and instead of pretending like this is some kind of major flaw in your own professional development, you could treat it as a gap which could be filled by people who have the opposite view of difficulty.
Uh, what? Most humans find social skills much easier to learn than crunchy engineering skills. Good social skills are way more common in society than, say, deep knowledge of algorithms and data structures.
But it’s not a competition. As you say, programming is rarely a solo activity. Our projects can fail due to a lack of teamwork. Or a lack of technical skill. Or a lack of empathy with our users. Or a lack of leadership. Or a dozen other things.
We need (and need to value, depending on the project) all of this stuff.
This is a really important point that we don’t talk about enough, because everyone gets uncomfortable and feels judged. But I can’t help myself, and I think there’s a lot more to dissect here.
I think programming is slowly bifurcating. We’re growing into two disciplines.
One discipline requires deep systems knowledge. A CS degree is recommended and if you can’t reverse a binary tree on a white board then you won’t be able to read our code base. Developers in this camp work on rendering engines, Vulcan drivers and schedulers. They spend lunch telling stories about that one time we wrote a game engine in J. Or arguing about type systems.
The second group makes most of the software (measured by lines of code). And almost all the software humans actually interact with. They spend their time working around weird safari quirks, arguing about electron, and enjoying the sense of ease and mastery that comes from staying in a single ecosystem for a decade. Getting better at their job looks like getting better at talking to users and customers, making on point estimates and making sure the team releases on time.
The part that gets all of us in trouble is that the first skill set is (arguably) harder - and thus much more prestigious. Everyone wants to be seen as an electrical engineer, not an electrician. So people write “software engineer” on their resumes. And then cry foul when they get assessed as a member of that first group. “I’ve never needed to know about red black trees in my career. It’s stupid for anyone to ask about it in an interview. Everyone just crams leetcode. JQuery isn’t flashy but it works and I can solve real problems with it. Hire me you fools, and I’ll show you!”. They’re sort of right. We’re just collectively confused about what sort of job they’re being hired for.
I think this confusion hurts everyone. I once wore my fancy engineering hat at a normal consulting gig. I coded up a complex solution to a tricky problem they had and called it a day. After I left, nobody on the team could understand or maintain some of the code I wrote. That caused huge problems for my client after I left. I think they might have regretted bringing me on board at all.
But we can’t even talk about this because we don’t have the language. We treat application developers like they’re just junior, in training versions of deep CS systems engineers. But practicing building kick-arse iOS apps doesn’t make you magically good at data structures. And vice versa - doing performance work doesn’t make you good at working with clients. (Ask me how I know). The majority of professional programmers do plumbing / user facing software. We need to destigmatise this stuff so people can be hired explicitly for these roles, based on the right set of skills.