Why? Because small incremental improvements cannot get published in high impact journals, and high impact journals are the currency to scientific prestige/grants/tenure.
Then someone (Newton, Darwin, Einstein) comes along and assembles a new framework based on new observations/theories that do a better job of explaining things. Eg the switch from creationism to evolution, or from newtonian physics to relativity.
Here are some more examples:
https://en.wikipedia.org/wiki/Paradigm_shift#Examples_of_par...
I guess what I'm trying to say is that the incentives are strongly biased to what can be easily seen as a increment on what has been done before, and that this is biased against work that may draw on what has been done before but is presenting a fairly different view to what is currently accepted.
Could Darwin have presented his view as an incremental extension of some existing view? (I am aware there were existing theories of evolution, but there weren't any like Darwin's or Wallace's to be built upon).
Einstein didn't get funding, i.e. an academic position. He did his initial groundbreaking work while having a day job in a patent office. He published his four groundbreaking papers in 1905 and was offered a lecturer position only on 1908.
Tenure with reduced teaching responsibilities is the way big thinking gets funded these days.
EDIT: I would also argue that developments like the printing press and the world wide web would have been difficult to justify as incremental extensions of existing work.
I'm not saying these developments came from nowhere, nor that they didn't build on previous work. I'm specifically arguing against the idea that it's fruitful to just focus on work that would be seen as incremental improvements on existing work.
This could be normalized within fields, but in practice it's not. For example, the NIH K-awards have a fixed eligibility period, which seems to keep shrinking.
With an increasing number of researchers isn't that a great thing? If you publish your little incremental innovations then everyone can build on them. If you try to build your giant all-encompassing framework before you publish any of it, you'll have to do it all yourself (if it even amounts to anything).