I mean, this is what evolution does too. The variants that 'looked right' but were not fit to survive got weeded out. The variants that were wrong but didn't negatively affect fitness to the point of non-reproduction stayed around. Looking right and being right are not significantly different in this case.
many corollaries exist. "looking right" is not at all General Artificial Intelligence, is my claim yes.
That this strategy is apparently enough to convince a large number of (supposedly) intelligent people otherwise is very troubling!
Not saying that General AI is impossible, or that LLMs couldn't be a useful component in their architecture. But what we have right now is just a speech center, what's missing is the rest of the brain.
Also, simply replicating / approximating something produced by natural evolution seems to me like the wrong approach, for both practical and ethical reasons: if we get something with >= human-like intelligence, it would be a black box we could never understand how any part of it actually works, and it might be a sentient being capable of suffering.
secondly, people selling things and people banding together behind one-way mirrors have a lot of incentive to devolve into smoke-and-mirrors.
Predicting is a social grandstand in a way, as well as insight. Lots of ordinary research has insight without grandstanding.. so this is a media item as much as it is real investigation IMHO