The best solution I’ve seen to this (if you have the time, and are interviewing for a variety of roles) is to have the same code (& same bugs) in a variety of languages. And let the candidate use their own computer and their own tools to work on the problem. If you’re hiring for a python role, get the candidate to debug python code!

I’ve done hundreds of interviews like this, and it’s fascinating watching what people do. Do they read the code first? Fire up a debugger? Add print statements and binary search by hand? I had one candidate once add more unit tests, to isolate a bug more explicitly.

After hundreds of interviews I still couldn’t tell you which approach is best. But if there’s one trend I noticed it’s that more senior people (& older people) seem to do better at this assessment. Which is fascinating. And that implies it’s not simply a test of what tools the person is familiar with most recently.

As for luck, I agree this is a problem. It’s mitigated somewhat by having a bunch of easy bugs to fix instead of one hard one. But even a debugging problem like this should be one of a set of assessments you perform. If you fail 5 small assessments in a row it’s probably not luck.