No human maybe, but could an AI?

I fed the prompt for https://leetcode.com/problems/two-sum/ to ChatGPT, which generated code for an answer, and I then asked it for additional test cases, which it gave. What's mind blowing though is that it "reasoned" that:

> These test cases demonstrate that the twoSum() function can handle cases where the input array contains negative numbers, where the target is negative, where the indices of the two elements are not adjacent, and where the input array contains duplicate elements.

Full transcript: https://pastebin.com/BpdcuWEM

The problem is that according to the post at the root of this thread the AI would have written a test to verify that A < B < C whereas it should have written a test to verify that A < C < B. If the AI or a human had also written code to match A < C < B then nobody might ever know until the day that condition comes up when the system is running.

I can imagine that it would be stressful to review code like this. Where you know you cannot trust a single line of it but that it looks consistent. You might have to verify back to the original assumptions every time and and make sure that you have not been led astray by the AI.