Yeah, with my interaction with ChatGPT it just spits out general concepts that almost make it seem like it exhaustively covers the prompt, but from those concepts sometimes there'll be 1 or 2 that just won't be applicable to the reality of the real world. Large language model lacks the acuteness to anticipate any unintended consequences and that 1% of edge case is where things might truly matter.
ChatGPT lists (common) downsides, remedies and situations when to avoid unit testing. Do you have the expected answer in mind, and could chatGPT have possibly deduced this from your prompt?
My problem and solution is probably too wide to explain here, so let's take an easier example.
TDD where you write the test before the code, is a widespread concept now. Imagine a time before this was the case. Would ChatGPT be able to come up with such a concept? Probably. Would it be able to really understand all the consequences of such a setup? I highly doubt it. Even if it did, would we trust it?
In my opinion, such things can only be tested in the real world, where you work with real projects, developers and testers, and gather all the impacts. I don't think a LLM could derive such things.
Re:91% of ML Models Degrade over Time
by throwaway4aday on Friday April 14, 2023 @07:53AM
(#35568421)
If you can't explain it in a comment then how did you explain it in the prompt(s)?
For example, as a senior software developer, I'm not convinced pure unit testing is always the way to go. I was thinking about writing my own alternative and how to weigh the benefits against the drawbacks. I was querying ChatGPT about it, but it's just able to restate the current "common knowledge".
So yes, ChatGPT can come up with new concepts. It however, cannot find out the problems, issues and unexpected consequences of applying it in the real world. Somehow, these things will have to be fed into it again, in either the form of text, images or video.
99% of current writing can be replaced by ChatGPT. But this 1% of things only people with real world experience can come up with, it cannot do.