But I'm not sure that it is an inevitable fact that most people will end up having the same ideas. Someone once said that in humongous countries like China and India, it is statistically inevitable they have thought of most of the things the rest of the world have thought of.
But why hasn't China come up with thoughts that German philosophers have come up with, or with specific ideas that have come out of the west? I think it's because there wasn't a focus on the things that mattered to Europeans at the time. Culture and environment matters, and dictates the ideas we are capable of having.
Large populations are not made up of uncorrelated, independent individuals. Most of the those individuals have highly correlated thoughts, mostly driven by the culture at large.
Either everyone else is stupid, or too lazy to come up with a plan to do it, or a myriad of other reasons. It doesn't matter why.
Like someone else pointed out the industry timing, state of technology and funding for that matter all drive these possibilities.
In the end only execution matters, and that comes with a long list of prerequisites aligning just right to even begin to form a possible positive outcome. It's luck and determination. The rest fail anywhere in-between, only to try again a decade or so later when the cycle repeats.
Welcome to the status quo. Don't like it?
Change it.
If researchers are not asking this themselves then they should be drummed out, and if they are asking this and they don't have an answer, and are still doing it, then they should be drummed out. Good people know why things didn't work last time, and they can give you a pin point explanation as to what needs to be done to break the bottleneck and make progress.
Case in point : Hinton and Deep Networks. Training big networks is impractical, you need vast amounts of labelled data, you need vast amounts of compute, the over fit. So - build auto encoding layers, build rectified activation functions, do drop out. Well developed, careful attacks on specific issues.
Another one is what's happening with fusion research at MIT right now : Whyte's attacking scale, attacking containment, attacking ablation. There are specific problems articulated, specific approaches to tackle them. These approaches come with risk, large risk, but it's not "because this time it's different"
But the fact is, an idea's current feasibility may be a function of its present constraints. Opportunities open up when constraints change [1]. This is why it can be useful to revisit old ideas and test them against the current environment.
[1] "Objectives and constraints", https://www.johndcook.com/blog/2018/06/26/objectives-and-con...