So at least in my area there is no research career. It's a dead end.
1.) I am sorry that you are going through this.
2.) Your experience perfectly matches that of a good friend of mine. She has been doing excellent work, she has had some great pubs and yet she considers getting tenure to be about as likely as winning a lottery.
I wish I had something tangible to offer, but the best I can offer is my good wishes. Seriously, best of luck!
For biomed stuff, it's frustrating that the NIH seems like it could address this problem (they control a huge proportion of the available funding), but have no interest in actually doing so.
If I were king for a day, I'd seriously consider shrinking the training grant pool and redirecting that money to developing viable career paths for "staff scientists" or other experienced-but-non-PI employees.
Giving money to staff scientists would work against both of these benefits. It would cost more and could reduce the number of trained scientists entering industry. (Although that isn't clear. Creating more NTT positions might just give hope to more of those eternal postdocs.)
I agree, but it certainly does not feel like the NIH has thought about this carefully either. My impression is that the NIH does what does because this is how it has always been done (and that worked well enough for the PIs they consult). I would love to see data suggesting that the current arrangement is optimal--or even close to it. The only thing that comes to mind is a recent paper arguing that peer review scores for grants were mildly predictive of the number of number of resulting publications.
As for staff scientists, one good staff scientist could certainly be more productive than a gaggle of newbie masters' students on their own. Right now, everyone is incentivized to do just enough to get the next paper. Permanent employees could build up infrastructure (protocols, code, etc.) that is actually reusable and ultimately more efficient.
Look at UCSF, they don't even have an undergrad program.
Were it not for the historical accident that we do things this way, I strongly doubt anyone would suggest we adopt it for whatever tiny educational benefit it may impart to undergraduates -- most of whom won't even go into the academic fields they are studying.