One tendency I've noticed is that LLMs struggle with creativity. If you give them a language with extremely powerful and expressive features, they'll often fail to use them to simplify other problems the way a good programmer does. Wolfram is a language essentially designed around that.

I wasn't able to replicate in my own testing though. Do you know if it also fails for "mathematica" code? There's much more text online about that.