Nobody knows. That's the long and short of it. If you find any set of traits in common among tenured prof's, and look for those traits in the grad student body, you'll find a roughly 10x surplus of people who are good enough to be in the pipeline.
Disclaimer: I was not one of those people. I got my PhD in 1993, and knew that I was clearly not professor material. So I steered myself into industry.
As for experience 30 years ago, the oversupply of PhDs was already widely known at that time. We called it the "birth control problem," and it was discussed over the then-nascent Internet. I mentioned it to my dad, a chemistry PhD, and he said: "Oh yeah, we knew about that in the 1950s."
A perpetual optimism about career advancement is possibly just a social convention. At my workplace, the HR message is that anybody can move up. But we can't all move up, and none of us are ever told our realistic prospects.
By the way I'm not trying to play "internet tough guy" here, but am just trying to be realistic. I'm actually quite sympathetic about the problem.
I do. You need to either be outstandingly, obviously better than almost everyone else, or you need to be good enough and lucky. You can probably think of examples of both of these cases.
I'd add a couple of other factors: The people I know who got into tenured jobs, had binders full of shovel ready research ideas (backed up by literature and/or preliminary results), and a comprehensive knowledge of the literature in their specialties.
We do know exactly how change this, though. Since it's probably not affordable or sensible to triple the number of profs (you can't do this without making profs much worse), the only way to increase the chances of the average PhD is to graduate fewer PhDs.
Back in the mid-20th century only 10-15% of people even got undergrad degrees and we did not have this problem so much.
Disclaimer: I'm a PhD, and have spent my career in industry. My dad did likewise, and my grandfather had a consulting business. I was no threat to anybody's professorial aspirations, and I was in grad school because I just wanted to have the education.
Should I have been kicked out, to make way for more worthy students?
Also, limiting the degree could backfire, because it could end up driving away the best students along with the worst, while attracting mediocre students who figure out how to play the game.
I would certainly welcome some reforms to graduate education, but I don't think that the PhD degree can be turned into a guaranteed career path.
Maybe. Does the astronaut program drive away the best and attract mediocre people?
I'm being extreme to make a point. I don't think high standards drives away the best people.
> Should I have been kicked out, to make way for more worthy students?
I don't understand the point. You competed with others for your spot, right? If that was working properly, the more worthy would have taken your spot. Sounds like you made the cut.
I think we'd agree that the problem is in expectations of students going in. I agree with others in this thread that profs have a duty to help people make realistic decisions. And as long as some subjects are set up to rely on a badly-paid middle class of postdocs, they are missing the incentive to do so.
In AI and robotics right now, it's very hard indeed to get a good postdoc since their opportunities are so good. I'm a prof and I'd love to exploit some but they all make more money than me. Good institutions are routinely hiring AI people straight out of PhDs. The problems we are talking about vary enormously by field because the supply and demand are wildly out of whack in some fields.
I made the cut, but to reduce the number of PhDs by (say) a factor of three, I might have been cut at some point. That could have been at the entrance to grad school, or somewhere else down the line, up to being told not to bother submitting my dissertation.
One idea is for funding agencies to set a minimum salary for post-docs, say $100k/y. Doing so would automatically change the number of post-docs and the standards for hiring them, while letting those of us with no academic career aspirations do our own thing.
I suppose that supports your point. ;-)
In AI and robotics right now, it's very hard indeed to get a good postdoc since their opportunities are so good.
Maybe that's the solution. It means that the people you're training actually have realistic career options, and you don't have to create an artificial job market for them.
I've always known that the academic market is saturated. I still don't know what it actually takes, in practice, to penetrate.
I contend that this guidance failure stems from a policy that aims to inflate certain non-informative statistics.
More to your point, however, I think you over-estimate the quality of the information going around. Every college professor will tell you it's difficult, but they'll also guide you based on their experience from 30 years ago.