LLMs really do still just reassemble things in their training data. There’s just a lot of it now, people anthropomorphise and struggle visualising large things. Some people say it’s truly reasoning but hit a topic that is under represented in the data of any LLM and it’ll transport you very quickly back a couple of years and ruin the illusion quickly.
It could be that with enough tokens, big enough context window, and ability to dig out the relevant partials, many such thought processes could be simulated.
The necessary precursors to the counter example where definitively in the training set, otherwise the LLM wouldn't know how math works, but at the same time, we can't tell whether there were mathematicians who got 90% of the way, then gave up and the LLM just did the last 10%.
From another old comment, someone else was trying to find a counterexample with 16 variables using a computer to make thousands of attempts and failed. So it's far from obvious that the trick to add a variable solves the problems.
My guess is it was not in the training. Only the 2D "almost" countraexample was in the training, but it was not clear how to fix it.
It's not clear how much steering Levent Alpöge did to get the result. He said he did during the final match of the World Cup, but he is from Turkey and living in USA and the game sadly was quite one sided, so I guess he does not care too much. So my guess is that he had a long chat with Fable.
I'm guessing too much, but if I can guess one more time the problem was probably too difficult to get solved by Levent Alpöge alone and by Fable alone, and it's a genuine Centaur solution.