That's more or less my experience with code generation. To be fair, I haven't paid for ChatGPT-4, which I'm told is much better. Still, if I ask 3.5 a leetcode style problem I've done, it comes up with a good solution. If I ask it a leetcode style problem I just made up, it generally comes up with garbage -- the solution doesn't work, solves a different problem than I asked, or ignores a restriction I imposed.

The place where I've found it really handy is if I want to make stuff up. Things like "Give me a mundane event that could happen to a 25-year-old leatherworker for a story I am writing." or "What should I name a piece of software that allows people to collaboratively draw a landscape scene?"