ngomezdev
At my small company we interview all candidates, regardless of experience level, using the same set of questions during the primary round. They’re a combination of open ended scenario type questions and pure debate style questions. We try to design them such that when you’re answering them, it’s similar to when you’re on the job collaborating on a technical solution, or triaging an incident.
We provide limited information at first to force the candidate to immediately ask for more info, to see what kind of questions they ask and what kind of information they care about. When they start to provide the solution, we like to throw in alterations or wrenches to see how they pivot. It’s very much “turn based”.
If you’re taking it seriously, it’s very obvious when candidates are either solving it for the first time, are recalling true experience, or are simply reciting information like they studied it. That’s why many of our questions work for many levels of engineering.
Additionally, because there is so much back and forth, you get to also “vibe-check” the candidate engineering wise. In our experience most candidates have enjoyed the process, so we feel like we uncover a lot about personalities here as well.
Unfortunately, the heel to this approach is that you pretty much have to conduct these in person. Because an LLM hypothetically would do well in this style of interview. We’re hybrid one day a week so it works for us, but it doesn’t exactly scale.
Perhaps your workflows and processes are so efficient that this actually makes sense. But I’m still not convinced AI assisted development is at that point yet; I think the classic Build vs. Buy arguments still hold.
How much is this going to cost to maintain? How many dev hours? How many tokens? Is your in house solution going to provide the business and customer enough perceivable value that it’s worth the opportunity cost? Are you going to apply the same rigor and security testing that the popular frameworks are getting from the largest research firms in the world? How do you know the LLMs aren’t going to apply similar patterns and make the same mistakes? Why can’t you just use that AI expertise to accelerate patching so your exposure time is limited? How do you know you’re going to outpace the major frameworks and tooling providers in 2 or 4 or 6 years when they catch up with AI assisted development, if they haven’t already?
And very personally speaking, I’m still holding my breath to see if this AI bubble is a real thing or not. Because if it ever pops, well, that’s not going to go well for companies that are embracing this stance.