Have you been using 3.5 or 4?
I fed the problem into ChatGPT later and it was utterly unable to comprehend it, but confidently gave wrong answer after wrong answer.
In my own assignments however, I focus less on algorithmic stuff but more on adding and mixing several things together. E.g. instead of just sorting, do group & sort, and a combination of a bunch of other practical stuff like reading big-endian binary files.
Just for fun I asked ChatGPT 4 to calculate the RMSE between two vectors both in English and Portuguese (also translating RMSE to Portuguese) and it gave me the same code for both questions (asked in separate instances). It would be interesting to know what restrictions you applied.
I'd expect simple tasks like calculating RMSE to definitely be within the abilities of LLM, you might combine things like actually reading the vectors from a CSV file (or a custom format) and calculating RMSE then sorting them etc to see the limitations of LLMs. Most students have no issues with calculating RMSE, they have issues with trying to do all the other stuff that leads to it, and then the combination of sorting and other tasks.
Regarding the restrictions, most of them are just don't use itoa/strtod or strcpy or some other standard library functions.
I agree with you, in my experience, ChatGPT is a better search engine but it is not capable of composing the various parts of an application in a cogent manner. I also think that the current UI is not appropriate for software development and I am sure there are efforts going on to create something closer to Jupyter notebooks for programming. That may be a game changer for your students (and you).
I don't think Jupyter notebooks or like similar REPL interfaces will help too much for my course, at least in the current syllabus. I'm aiming to teach about pointers, memory management etc, the more fundamental parts of how to interact with computers instead of a high level language. Though I would agree that the current UI is suboptimal, some improvements in allowing students to visualize memory layouts and see how their code manipulates memory will help a lot.
I can't recall exactly but I think https://godbolt.org/ might do that for example?
Godbolt is a compiler explorer, it shows disassembly of a code but there's nothing to visualize each step in the process.
For my own course, I think several factors contributed to students not utilizing ChatGPT as much:
It's difficult to say if advancements in LLMs will make my job harder, where say copy/pasting my more complicated assignments can lead to correct results. But from what I can see right now, LLMs still have trouble solving novel problems, so it's probably always possible to come up with assignments that's difficult for them to solve.