Potentially not great.
If you look at the AIME accuracy graph on the OpenAI page [1] you will notice that the x-axis is logarithmic. Which is a problem because (a) compute in general has never scaled that well and (b) semiconductor fabrication will inevitably get harder as we approach smaller sizes.
So it looks like unless there is some ground-breaking research in the pipeline the current transformer architecture will likely start to stall out.
Also in general, I have yet to see these models plateau, Claude 3.5 Sonnet is a day and night different compared to previous models.
More compute hasn't been the driving factor of the last developments, the driving factor has been distillation and synthetic data. Since we've seen massive success with that, I really struggle to understand why people continue to doomsay the transformer. I hear these same arguments year after year and people never learn.
That being said, the ARC-agi test is mostly a visual test that would be much easier to beat when these models will truly be multimodal (not just appending a separate vision encoder after training) in my opinion.
I wonder what the graph will look like in a year from now, the models have improved a lot in the last one.