Fair enough, but my bet is that even trivially recontextualizing an interview question is enough to wreck an LLM's performance.
If you change the "find a word in a grid of letters" question to a "find a Collatz sequence in a grid of numbers" question, does it still work? As an interviewer, I would expect a qualified candidate to spend maybe an extra 5-10 minutes asking clarifying questions and understanding the difference between the two.
I was talking purely about those abstract interview questions where the full problem can be summarized in a few lines and the expected solution is ~50 lines max. I've just tried with Claude 3.7 and it's produced a seemingly plausible solution at the "word finder" problem (2D grid of letters - given a word, return if it's present in the grid as a series of adjacent cells in any direction).
I agree that they're far from taking our jobs, which is my point - those interview questions are absolutely not representative of what we really do and optimize for the wrong outcome.