A mid-level human engineer can iteratively fix the mistakes they start making, instead of that just being the final result. Can GPT-4 do that?

Yes, like I said if you give it the results, eg "That code resulted in this error. What's wrong?" it will more often than not correct its mistake.

Hooking it up to automatically run the code in question and examine the output is a trivial undertaking - many folks have already done this.

My experience has usually been that it will apologize and then produce another snippet with a different flaw. When that is pointed out, it will usually go back to the original snippet with the original flaw. It sometimes also insists that it has “run the test suite and they all pass”, just like a human programmer that is trying to fake it until they make it I guess?

It does it pretty well for Python with code interpreter, especially for quant/ ML heavy code.

I wonder if anyone has done a feedback loop with GPT and compiler errors to train this sort of thing

they have, with langchain

I would peg gpt as between junior and mid. Mids (around 5-10 years of experience) tend to produce overly complex code for the problem at hand.

Dude, chatgpt as a junior or mid level programmer? Have you actually met junior or mid level programmers?

I’ve got 20 years of experience in the tech industry and chatgpt is far far better than any mid level engineer I’ve ever met and I include FAANG engineers too.

Chatgpt can answer any leetcode programming question in essentially every programming language in existence in less than 20 seconds.

Chatgpt can analyze classes for errors written in every language in seconds.

Chatgpt helped me write animation, ai, sound handling code in c++ and helped me build a full working game, the total code generation time for chatgpt “thinking” was on the order of one hour.

I don’t know any senior engineer who could help me (a c++ newbie) write a whole working game engine with all the necessary systems in less than one hour of “thinking time”.

I’m starting to think that all the “naysayers” of chatgpt simply used it once or twice, saw one error and dismissed the whole thing. Or they’re simply afraid for their own skills and avoid chatgpt as the alternative scares them too much.

Can you share a link to the game you made with ChatGPT?

Yes, we employ them regularly. And yes, I use both ChatGPT and copilot regularly.

In my experience, if you compare the unedited output of ChatGPT to the unedited output of someone with a couple years of experience in the given domain, ChatGPT will have more subtle bugs than the human provided output.

Hence my between junior & mid assessment. It doesn't mean that it's useless though. Having a tool at that level of experience for every domain in the world that can cook up code near instantaneously is damn useful. And I assume it'll only get better.

> I think GPT4 right now is about on-par with a mid level developer.

At writing code, that might be a slight exaggeration. At explaining code it's at least at mid level, possibly better.

How many mid level developers do you know that understand the syntax and apis of every single programming language in existence?

I’ve used chatgpt recently to build an entire c++ game with sound, keyboard and mouse control, ai, etc, having never used c++ before. I’ve also never built a game before. I’ve worked with a lot mid level developers and I’ve never seen any that can write me a a c++ file giving me working code for a collision system in 30 seconds. Let me know if you find one though.