If you could do a perfect job of training a GPT on the entirety of published academic literature, the total of what that GPT could spit out would be limited by the knowledge contained in academic literature. At the same time, you'd have created a tool that is cheap and does a good job of synthesizing knowledge/answering questions across all disciplines. The model will never replace the scientists who are working at the very bounds of their fields, but it doesn't have to in order to be extremely useful, even useful enough to replace a majority of knowledge workers.
Just because GPTs can't be smarter than the smartest humans doesn't mean they can't be smarter than most humans.
Or perhaps the insight isn't trivial but follows analagous reasoning from some other obscure result. Or perhaps a million other things.
I don't mean to claim GPT will do this. I just mean to point out it can't be fully excluded GPT is able to.
An astonishing amount of what we each believe is plain wrong, especially when we have little stories about it.