> I feel like take home tests are meaningless and I always have. Even more so now with LLMs

This has been discussed many times already here. You need to set an "LLM trap" (like an SSH honey trap) by asking the candidate to explain the code they wrote. Also, you can wait until the code review to ask them how they would unit test the code. Most cheaters will fall apart in the first 60 seconds. It is such an obvious tell. And if they used an LLM, but they can very well explain the code, well, then, they will be a good programmer on your team, where an LLM is simply one more tool in their arsenal.

I am starting to think that we need two types of technical interview questions: Old school (no LLMs allowed) vs new school (LLMs strongly encouraged). Someone under 25 (30?) is probably already making great use of LLMs to teach themselves new things about programming. This reminds me of when young people (late 2000s/early 2010s) began to move away from "O'Reilly-class" (heh, like a naval destroyer class) 500 page printed technical books to reading technical blogs. At first, I was suspicious -- essentially, I was gatekeeping on the blog writers. Over time, I came to appreciate that technical learning was changing. I see the same with LLMs. And don't worry about the shitty programmers who try to skate by only using LLMs. Their true colours will show very quickly.

Can I ask a dumb question? What are some drawbacks of using a hash map? Honestly, I am nearly neck-bearded at this point, and I would be surprised by this question in an interview. Mostly, people ask how do they work (impl details, etc.) and what are some benefits over using linear (non-binary) search in an array.

"Drawbacks" was the wrong word to use here, "potential problems" is what I meant - collisions. Normally a follow up question: how do you solve those. But drawbacks too: memory usage - us developers are pretty used to having astronomical amounts of computational resources at our disposals but more often than not, people don't work on workstations with 246gb of ram.

I think the better word is tradeoff since there are no perfect data structures for each job. The hasmap has the advantage of O(1) access time but the drawback of memory usage, an unsorted nature and the depends on a good hashing function to minimize collisions. A vector is also O(1), but it has an upfront memory cost that cannot be avoided. A map has a O(Log(n)) access cost, but has less memory usage, is sorted by nature and the comparison function is easier to implement. Three similar data structures, but each with its own tradeoffs.

Good point about collisions. When I wrote the original post, I didn't think about that. As a primarily CRUD developer, I never think about collisions. The default general purpose hash map in all of my languages is fine. That said: It does matter, and it is a reasonable topic for an interview!

If you really need to test them / check that they haven't used an LLM or hired someone else to do it for them (which was how people "cheated" on take-home tests before), ask them to implement a feature live; it's their code, it should be straightforward if they wrote it themselves.

This is an interesting point. During the code review, you can add a small twist to the problem, then ask them how they would modify the code.

What if I use an LLM but I understand the code?

The drawback is that elements in a hashmap can’t be sorted and accessing a specific element by key is slower then accessing something in an array by index.

Linear search is easier to implement.

These are all trivial questions you ask to determine if a person can develop code. The hard questions are whether the person is the cream of the crop. The amount of supply of developers is so high most people don’t ask trivial questions like that.

    > What if I use an LLM but I understand the code?

That's OK. I wrote: <<And if they used an LLM, but they can very well explain the code, well, then, they will be a good programmer on your team, where an LLM is simply one more tool in their arsenal.>> If anything, I would love it if someone told me that they used an LLM and explained what was good and bad about the experience. Or maybe they used it and the code was sub-par, so they need to make minor (or major) changes. Regardless, I think we are kidding ourselves if people will not make (prudent and imprudent!) use of LLMs. We need to adapt.

If you are evaluating how well people code without LLMs you are likely filtering for the wrong people and you are way behind the times.

For most companies, the better strategy would be to explicitly LET them use LLMs and see whether they can accomplish 10X what a coder 3 years ago could accomplish, in the same time. If they accomplish only 1X, that's a bad sign that they haven't learned anything in 3 years about how to work faster with new power tools.

A good analogy of 5 years ago would be forcing candidates to write in assembly instead of whatever higher level language you actually use in your work. Sure, interview for assembly if that's what you use, but 95% of companies don't need to touch assembly language.

> ... LET them use LLMs and see whether they can accomplish 10X what a coder 3 years ago could accomplish...

Do you seriously expect a 10x improvement with the use of LLMs vs no LLMs? Have you seen this personally, are you 1 10th the developer without an LLM? Or is the coding interview questions you ask or get asked, how to implement quicksort, or something?

Let's make it concrete, do you feel like you could implement a correct concurrent http server in 1/10th the time with an LLM than what you could do it without? Because if you jut let the LLM do the work I could probably find some issue in that code or alternatively completely stump you with an architectural question unless you are already familiar with it, and you should not be having an LLM implement something you couldn't have written yourself.

> implement a correct concurrent http server in 1/10th the time with an LLM than what you could do it without

Absolutely fucking yes.

In that case, could you begin proving that point by having it write an http request parser. Let's make it easy and have it require content length header and no support for chunked encoding at first. Csn pick any language you like but since thats such critical infrastructure it must export a C API. Let's also restrict it to HTTP 1/1.1 for the sake of time.

Concidering this would probably at most take a days work to get at least a workable prototype done if not a full implementation, using an AI you should be able to do it in a lunch break.