Re:AI hype is built on high test scores. Those tests are flawed
by jiggawatts on Tuesday October 10, 2023 @11:43PM
(#37840625)
> valid end state most of the time
I disagree. Don't assume "most humans" are anything like Silicon Valley startup developers. Most developers out there in the wild would definitely struggle to solve problems like this.
For example, a common criticism of AI-generated code is the risk of introducing vulnerabilities.
I just sat in a meeting for an hour, literally begging several developers to stop writing code vulnerable to SQL injection! They just couldn't understand what I was even talking about. They kept trying to use various ineffective hacky workarounds ("silver bullets") because they just didn't grok the the problem.
I've found GPT 4 outperforms median humans.
Gpt-4 with help from a competent human will of course do better than most humans, but that isn't what we are discussing.