News for hackers. Stuff that matters.
Re:Fun and dystopia with AI-based code generation using GPT-... by username90 on Thursday June 24, 2021 @06:47PM (#27624923)
I think the "turing test" for code generation would be to be able to do most leetcode problems and other competitive programming problems. If you can do that you have done something amazing, and the dataset and testing for it already exists.
Re:Fun and dystopia with AI-based code generation using GPT-... by meetups323 on Thursday June 24, 2021 @08:43PM (#27625806)
Not really helpful, given they all have many publicly available solutions that could have been easily memorized (I'd be surprised if such solutions weren't already part of their training data).

More interesting would be to give it open tickets on popular OSS software, have the maintainers point to the file(s) where the fix would happen, and let it craft a patch.

Re:Fun and dystopia with AI-based code generation using GPT-... by nonameiguess on Thursday June 24, 2021 @08:42PM (#27625805)
Leetcode posts the answers. That could be accomplished by just scraping them. [1] Turing test would be give some vague, underspecified requirements for a system that does not yet exist, and implement a version the requirements writer will accept. Compilers and compiler generators have long been able to generate great code from well-written specifications and no one thinks of them as passing a Turing test.

[1] Of course, to some extent, that is what a GPT model is doing anyway. It's able to generate reasonable passing code given just a function prototype because it has scraped and looked at billions of examples of implemented functions with similar prototypes.

Re:Fun and dystopia with AI-based code generation using GPT-... by gajomi on Thursday June 24, 2021 @08:43PM (#27625812)
This would be the opposite of a Turing test though, since most people wouldn't be able to do this.