I've noticed this with Gemini recently - I have a task suited for LLMs which I want it to do "manually" (e.g., split this list of inconsistently formatted names into first/given names and last/surnames) and it tries to write a script to do it instead, which fails. If I just wanted to split on the first space I would've done it myself...