Like in mathematics or other tests students are not allowed to use the internet or an advanced calculator in order to test wether they truly comprehend the stuff.
If you think that your FAANG-interviews are any good, then just keep them in the format you already have, by making sure applicants cannot use AI during the test. I would have on-sites with pen+paper, whiteboard or a prepped/supervised machine, whatever.
Of course, applicants could use AI to train for the interview, but that is not a problem, as long as you test their comprehension.
My money is on "hurt".
I hope companies (including mine!) see the writing on the wall and stop trying to fight the future.
This isn't accounting for even just pretending we are testing candidates on anything remotely indicative of on-the-job performance.
In a few years, ChatGPT will be conducting these interviews.
Agreed.
I think developers who don't will be rare in a few years time. Just like developers who primarily rely on assemblers (versus compilers) have basically become extinct. Like we sometimes, very rarely, need to inline some assembly, we will sometimes need to use the ol' gray matter to figure out a novel algorithm or something.
I believe that avoiding learning these tools could be a existential issue for your present-day job. You don't have to come to depend on them, or use them daily, but you do need to understand how best to use (and not use) them.
Ever since the emergence of leetcode style interviews I've been shocked at how many people can reproduce leetcode examples, but still fundamentally have no sense of algorithm design outside of the context of a job interview.
Programmers with their sights set on acing a FAANG interview will just keep repeating leetcode problems until they start to memorize the common patterns (not the problems themselves of course, but the structure of these type of problems). What's disturbing to me is that I recall far more interesting discussion about algorithms in the era before leetcode dominated everything.
The common solution isn't to understand algorithms better, but to become a leetcode solving robot.
So it's no surprise to me that AI can pretty easily replicate humans that have tried to turn themselves into robots.
We see similar patterns in the art that AI can create. It's very good at replicating a kind of art style of designers trying to turn themselves into design robots.
What sort of job leads you do design algorithms?
> So it's no surprise to me that AI can pretty easily replicate humans that have tried to turn themselves into robots.
asking "here's a chatgpt solution to the problem... what's wrong with it?" would be a solid process imo.
But even more, it seems likely someone is going to be willing to supply LLM access at not that much more than the cost of computation.
I know OpenAI's business is selling access to their stuff at a premium but since current AI is not much more than brute-forcing of massive public data, it doesn't seem like this premium will be justifiable for long.
Particularly, ChatGPT can give apparently correct solutions that are wrong in subtle ways.
Also reading, understanding, reasoning about, and fixing code other's wrote is way closer to on-the-job performance.
Ambiguity that requires follow up questions for successful isn't going to be addressed by something focused on solving a problem that "thinks" it has all the information to solve the problem.
That's impressive perseverance. Did that wear on you?
From my side of the table I can confirm that of course luck plays a role. Nevermind the variability of the coding problems themselves, but the variability on the interviewers. Even in my company, that has a formal process and pretense of objectivity, interviewers quality and expectations vary greatly.
In short, I'd say what people on the hiring side hate to hear, but interviewing is ultimately a numbers game. I wouldn't take rejections too seriously. The best you can do is to improve your chances with proper prep (leetcode, system design) and interview broadly.
This sounds like yet more evidence that it's really just pledging a frat.
Thing is, in a small way, I'm trying to change things from within. I try to make the process as palatable and fair to candidates as possible, while working within the constraints of the system. When I train new interviewers some of my top tips are:
a) The purpose of the interview is to determine whether the candidate is a good fit for the company, and the company is a good fit for the candidate. b) The interview is an imperfect proxy for this.
It then follows that interviews shouldn't overindex in the coding round. If I have helped someone to get an offer that wouldn't otherwise, I'm satisfied.
BTW that's also one of the reasons I'm publishing this. I hope to push the point that leetcode-style coding interviews are outdated and should be burnt to the ground.
You're the person they're trying to fool, so if they cheat successfully you would never know. You've never seen overt displays of bad cheating, which is different.
Think you haven't seen a stick insect in years? Likely you have, but just didn't notice it...
Cheaters will always be more motivated than those trying to detect them - because everything is on the line for them.
I don't think it's widespread at least, since in my experience people that do well in technical zoom interviews do not drastically decrease in apparent competence when we move to in person rounds.
People definitely get told interview questions by recruiters though, if you count that as cheating then it is everywhere.
option c makes me nervous, but right now I can't see an ai correctly dealing with the ambiguity of the data sets i typically look at.i do a lot of "asking for clarification."
Interviewing is going to be shaken up but I think some methods are more timeless. I've been giving the subject code instead of asking them to write it for a while now. Largely because I wanted to talk more and watch them type less.
We start by discussing the coding challenge guidelines and I leave them vague. They need to understand the goal and what I've left out and suggest those guidelines themselves. Once we agree though, I give them the code. "Here's what's running now."
Then I update the goal and we discuss changes. I get them to "whiteboard" certain things, like what a query looks like with their proposed changes or whatever, and we discuss big-O, etc. This, imho, is how whiteboarding is actually used - not to write whole programs but to provide examples and pick them apart.
I feel that this would work even if they were using an AI in another window. We're trying to select for developers with common sense and domain knowledge, and who can clearly discuss engineering tradeoffs. Actually making the changes (the coding itself) was a big part of the job and that's decreasing, but imho all the other requirements remain. They'll still need to know how to handle the issues I talked about above, of trying to get the model to write the right code!
The real benefit of AI is somewhat shown in this paper, it effectively solved the problems through brute force generating millions of possible solutions. For real world problems it would be interesting to let GPT generate a bunch of different solutions and push them into a test environment and see which works best.
The biggest problem I see is black swan events where AI coded systems work great until something goes wrong and no human truly knows how all the pieces fit together.
Nevertheless the point is moot. I've invented completely novel questions (promise!), and saw them leaked online after asking them twice. The process is fundamentally flawed and large language models are just making that glaringly obvious.
This is by far my favourite question, the one closer to on-the-job coding. It also lends itself well to deep conversations with candidates.
But alas I don't own the process and still have to work within the parameters of the company. Whenever possible I ask this kind of questions, but other interviewers will default to leetcode-style rounds.
That's great! For now. But tomorrow's coming fast.
Interestingly when I asked a follow up, a harder version of the same problem, ChatGPT spits out code that sort-of looks correct, but is actually nonsense.
So, would it pass the interview? That's difficult to say. Interviews don't happen in a vacuum, you also consider the candidate's thought process, explanations, alternatives, tradeoffs...
Still, I see this is a game changer for this kind of interviews. So long as candidates understand and explain the output code, there is a good chance they would clear the interview. Even if the code is incorrect, it might given them some hints towards the right solution.
So where do we go from here? I always loathed this interview format, and these languages model reinforce it even further. Interview cheating has always been there, but it is generally so rare that it isn't a real concern. However these tools are too effective and easy to use. I can see a real divide between people who use them and people who doesn't. This type of interview might become even more useless at telling good coders apart.
My take is we have 3 choices:
a) Ignore it.
b) Try to fight it.
c) Embrace it.
a) is not an option. b) would make an already pretty dreadful process even more intolerable. My money is on c)
I can envision an interview format where we allow, or even encourage people to use ChatGPT and AlphaCode during the interview, much like you would use your IDE or a search engine. In fact seeing how a candidate understands and uses those code snippets can be a very interesting data point.
Either that or scrap leetcode-style interviews altogether.
P.S.: I was thinking about writing a blog post about this, if people think it'd be interesting.