I don't think anyone expected LLMs to do this well on code problems. Just a few years ago they were struggling to get the syntax right. But since it does seem to work in many cases, it's only reasonable to explore its limits. But in the end, automated math & programming will most likely not be done using LLMs, but with purpose-built systems. The major players are of course already working on this, including OpenAI [1] and DeepMind [2].

[1] https://openai.com/research/formal-math (feb '22)

[2] https://www.deepmind.com/blog/competitive-programming-with-a... (feb '22)