A known limitation of word-at-a-time GPT models is that they can't backtrack, edit, or iterate on what they're doing. You can see these limitations when asking them to solve puzzles involving "term substitution". They can only do these up to a certain complexity, and then they run out of steam.
However, if you run their output back through for a second round, they can continue substituting!
Essentially, GPT models make mistakes, but can also recognise those mistakes and fix them on subsequent iterations. Much like humans.
Another factor is that the "temperature" setting means that the outputs have an element to them that's purely random. Combining multiple outputs "anneals" out this randomness by converging on output that is the most consistent. Essentially, it's like asking a team of people to brainstorm, and then combining their partial solutions into a complete one. Once again, very human-like!
This has been tested (semi) scientifically. Running GPT 4 across some puzzles or exams might yield a 60% success rate if you give it one chance. If you let it solve each problem by running it four times and then letting it combine its own outputs itself (not via human-selection!), then it will get up to an 80% success rate. See: https://www.youtube.com/watch?v=wVzuvf9D9BU
"If I jiggle this is behaves differently" is like... half of all science!
At the end of the day, better is better, and I don't think that can be hand-waved away.
That's like saying someone is not smarter because they just "studied harder" and that doesn't count somehow.
"Let's publish a bunch of measurements we took" might be rigorous and look like what advancing science would look like if you squint. But this is just gaming the publishing game.
A lot of science is just taking down measurements and recording the result.
It's not always a giant leap, some grand insight, or a world-changing theorem.
The busywork in between might have looked like science and occupied a lot of time and space. Relativity and quantum mechanics generated a lot of falsifiable claims that were eventually tested. Yet we have been building larger and larger colliders to "collect and publish data" and the best outcome so far has been a job program for engineers (useful and good) and the Web (useful and good). But we shouldn't confuse collecting data and engineering with advancing science.