There are still a lot of hard problems to solve, they're just cross-disciplinary.

Software is the infrastructure of the 21st century and if you step out of the tech bubble to things like material science, biotech, etc. you'll see that their infrastructure is garbage despite there being really big returns available.

Developers who can marry modern software to those domains should remain in high demand for decades.

Do you have any companies or spaces you would recommend where theres hard problems but low signalling/barriers to entry? I'm autistic AF (no joke) and learn new domains in a frenzy of 3 months of joy and digging but its not easy to prove/signal that.

The best one is state or local government. You’ll be able to make a lot of change and get tons of experience with a low barrier to entry.

Anything related to public safety (including transportation) probably needs the most tech talent.

So much this.

Spend a month with a tech stack of any passing complexity, even ones at $FAANGco...

We're nowhere close to solving all the hard problems.

There are still hard engineering problems to solve, they just aren't related to scaling out services as much (there's still optimizations and improvements to be made in that space, but not nearly as much). You can't seriously think all the hard engineering problems have been solved, right?

Give me some then! I see maybe some cool stuff in the move to ARM processors. I also don't have the resume or fame or background to get offered the chance to solve them if they're out there - in 2015 there were so many and the average level of competence was so low that people gave me a shot anyways.

I've been trying to learn more about the dark corners of the linux kernel and graphics drivers for now. But its not 2015 and I don't have 3 years of experience in embedded/systems/C++ so no ones gonna hire me for that stuff because my resume just says "generic backend dev lol".

> in 2015 there were so many and the average level of competence was so low

The average competence is still low, the technologies are just no longer new and shiny (they aren't mature either), they're just better understood throughout the industry. 2010-2015 saw huge cross-industry revolutions in terms of web adoption, and until the next "revolution" happens, we'll probably be in a lull in terms of innovation in the web app space. Of course JS/Node will continue to "innovate", but overall I think that portion of the industry has been successfully "commoditized."

Hardware engineering (drivers, embedded, IoT, etc.) will always have tough engineering problems, but as you said, it's kind of hard to break into that (though maybe not, I haven't tried). Quantum Computing and other experimental and uncharted spaces are good sources of problems looking for well engineered solution (source: I work in quantum computing, on the engineering side), but is a much smaller and niche industry.

> I see maybe some cool stuff in the move to ARM processors

I also agree on ARM, I think things will gradually shift towards ARM-based processor architecture in the server and desktop spaces. From the software side, most modern programming can be run on those processors without issue, though there's some work to be done on optimizing compilers (hard, not very rewarding).

> I've been trying to learn more about the dark corners of the linux kernel and graphics drivers for now

This is an area I'm also particularly interested in. If I wasn't getting close to burning out, I'd probably focus on it on the side a little more. AMD is behind, in particular, with regards to graphics driver optimization and toolkits for machine learning (nVidia is beating them out there, thanks CUDA). I'm not sure what their hiring is like, but there's been huge industry shakeups over the last few years from them in the processor space and they're swimming in money.

Machine learning is also a hot topic, and probably will be for awhile. Security engineering (especially in the context of Kubernetes) is an ongoing issue that needs more attention.

There's definitely others. I'd eventually like to find a problem that interests me and I see a market for so I could start my own company, but alas, I am not good at ideas.

Do you mind being specific on how there are no longer hard engineering problems to be solved?

Well beginning in 2008ish, we saw websites and applications that served the entire world, literally billions of users. That simply did not exist before then. The internet as a whole was a tiny part of peoples lives. All the stuff that went into solving that was genuinely new ideas in servers, databases, ops, data centre's etc. Eventual consistency went from being a joke to the only game in town. Distributed computing on large data sets went from a Google paper in 2005 on Hadoop to a widely used idea. Smartphones went from zero to iPhone to literally more than half the world has one. Devops became standard. Google went from just managing 1 or 2 datacenters to running AI to optimize the power efficiency of each one.

When you already have architecture that is capable of serving the entire world, when everyone has a smartphone all that's left in that space is efficiency/cost optimization. Whatever shiny new programming language or framework comes out, the bedrock of computing is solid (and the shiny new programming language is really mashing together decades old ideas which is cool if you're new to them but boring if you're not) . Every company is a mess on the inside of course, but they do their core functions well enough. AWS did not exist 10 years ago, now it's big internet drama when it goes down for a few hours twice a YEAR.A detailed postmortem comes out (not standard before), revealing the problem was a configuration change (because everything has long since been automated, so it's driven purely by configuration) .

I know not everyone is interested in those same problems, but I thought they were interesting and what's left now in the guts of software is an adoption phase, people rediscovering lessons that others long since figured out. Perhaps people who work on higher layers of the stack think differently, I'm only interested in the deep tech below, and it's pretty solid at this point.

There's still lots of relevant work to do deeper in the stack. It might just not be at large internet scale companies (I doubt this but lets assume it's true). Most every university has a systems research lab doing interesting things. I think I paged through 5 different professor's websites in CMU's systems department the other month and found 9-10 interesting projects.