In my somewhat limited experience, companies often treat 'senior devs' as team leads, building mentoring responsibilities into the role rather than treating them as 'highly skilled devs who work efficiently and do not require much oversight'.

For a mentoring structure, it is always more efficient to have more mentees than mentors, hence more junior roles than senior roles.

The 'senior' = 'superproductive' structure requires fewer employees overall, but also brings an element of risk since the work is more concentrated.

The net effect is fewer senior roles.

No, the thing is that there are not enough "interesting" and "complex" things to do - industry doesn't usually need to scale an app to the size of WhatsApp or Twitter, and even when it does, a handful of senior people managing a team of junior devs usually suffice. So, experience usually isn't really a benefit (after 3-5 years of experience), and can be a detriment for the reason you mention (boredom with doing the same CRUD app over and over again).

This is horrible. Maybe I should suck it up, but it's not my fault industry has zero imagination.

What are alternatives? I want to be in R&D, something that America doesn't seem to believe much in.

What? America doesn't believe in R&D much?

I'd push back rather hard on that statement. US companies invest loads of cash in R&D, more than any other country. The problem is that if you expect to be doing R&D for a YC startup (or even a typical SV consumer web or B2B startup), you're really barking up the wrong tree—those sorts of companies don't do a lot of R&D. If you're really interested in that kind of work, you'd be far better off in bioscience or hardware. Or at least larger, established software companies.

You need to be careful on how you define R&D. Pure R&D, the kind of culture that spawned say UNIX, Oak (Java) or Plan 9 etc is on the decline as in recent years the bean counters tend to look at pure R&D as a waste of money as there is no medium term return on investment. Look at the decline of IBM Research in the US as an example. Google and in some cases Microsoft are exceptions to this.

However the US tax code allows companies to write off new development of new systems as a tax break, so you typically would be doing this kind of R&D at a startup until the system moves into production and is then accounted for differently for tax purposes.

So US companies invest load of cash in R&D, just not the traditional kind of Pure R&D.

Yeah, this is what I was referring to. Pike lamented this in his famous essay, "Systems Software Research is Dead." In it's place, we tend to have more focused R&D, which is still kind of nifty, but can imply a certain amount of acquiescence with the status quo.

In software, Microsoft Research, IBM Research, Intel OTC, Qualcomm QuIC, etc. come to mind.

I have no idea. Google X, Facebook AI Lab? I'm struggling with this myself, the solutions (for me) now seem to be: learn something new every year at work (by changing the technologies your employer uses or changing employers), and research in the afternoon.

I did exactly this - did my research after work, when my employer didn't budge in terms of putting on new technologies, or even improving it, I changed.

The other move is to do a start-up and work on one's own R&D (how you do it is your call)

The problem is that I'm interested in robotics/AI, which (right now) seems to be a high-investment/low-probability-of-success endeavor.

On the contrary. I feel like if you can prove competency at depth in AI/robotics, you will be absolutely _set_ for the next ~half century of computing. The sort of problems we're solving in the areas of machine learning/planning have applicability literally everywhere, even outside of STEM fields. (looking at you, social sciences, a big data / machine learning twist is certainly in your future)

The hard part is the "proving competency" part. I consider myself a somewhat competent programmer, but trying to catch myself up on the state of the art in my focus; N body simulation; is a hard enough problem, and even when you feel like you understand it, how do you let other people know? (little bit of chicken in egg here, since normally the PHD is this sort of signalling.)

The answer I chose is "build fun projects in my free time." and with the advent of more accessible programmable robotics, you can certainly explore both of your interests. (I should take my own advice here too...)

Do we know what role Acton applied for? I saw the twitter posts about the rejection but never saw what job he was applying for. It's hard for me to generalize about the industry from this one specific case, without even knowing what job he was allegedly rejected from (his experience may have been undesirable for an entry level position where the hiring manager felt he was overqualified, or conversely he probably lacked the experience at the time to get the role of CTO).

He was VP of Engineering at Yahoo before he left, so it stands to reason that he was applying for a very senior role.

That's probably a safe assumption. I'd imagine he was looking for another very senior-level job. In that case, it seems silly to use it as an example for why there might be a shortage of talent for entry-level to mid-level engineers. It doesn't seem like his experience was detrimental, or that he didn't fit some specific pattern (based on age, etc).