The training set is full of code for these problems.

Put your query into Google and see how many thousands of answers appear.

Your tests are in the training set.

True but the horrifying truth is 95% of our actual problems at work are also in the training set...

My biggest problems are obscure bugs that I would hardly understand even if the AI would just give me the workaround.

This idea is why I wanted to see the code created by GPT at each step during this conversation for this post...

Oh wow. I'd love to see someone set up an environment where GPT solves a programming interview incrementally as a human would - with a teleprompter of describe-as-you-go text for the interviewee to read out. Couple it with speech-to-text for follow up questions and you can conman your way through any coding focused interview.

Not that I want or need this to pass interviews. But it would finally force the industry to find a better way to evaluate candidates.

Cheat on your Technical Interview as a Service™

I pasted in the question I typically ask candidates at $dayjob. It did better than the vast majority of candidates. (It’s a very open ended architectural question that’s specific to the industry I’m in.)

At the end I said “write it in Rust” and it wrote a plausibly good implementation.

I’m not sure I can trust remote interviews any more…

Actually it sounds like recruiting just got a lot easier for you!

That's close to answering a question from a bit over a year ago: "in 5 years will there be an AI that's better than 90% of unassisted working programmers at solving new leetcode-type coding interview questions posed in natural language?" https://news.ycombinator.com/item?id=29020401

It didn't get a very welcoming reception -- "highly unlikely", and the tenor of other opinions in those comments then tended the same way. But I was being too conservative. (I thought so at the time but it felt sort of outside an Overton window of reasonable technology opinions.)

What? Leetcode is purely about memorization. Of course a model trained on those solutions will regurgitate them. That how humans do it!

you missed the word 'new'

Same here. It passed the tests for backend / frontend and created perfect unit tests as well.

But if it's a remote job, won't that same developer do great by just delegating all their tasks to the AI, just like they did in the interview? Crazy stuff.

I asked my usual question and it got the answer wrong, but in a believable way that it could’ve learned on stackoverflow.

(Basically it tried to parse HTML with regexes.)

It's been an open secret on blind that cheating on remote interviews is highly, highly common, and has been with language models since late 2019. Best part is that these same folks end up getting promoted because as it turns out once you have passed the stupid coding hazing ritual, most jobs at the FAANG will NOT need you to solve leetcode style problems.

I'm honestly excited as heck that something will finally kill off this practice. Don't let the door hit you on the ass on the way out, leetcode.