I can't imagine this being an effective qualifier for any but the most junior of devs.
I’ve had a majority of candidates with five-year experience as a data scientist not be able to load a csv file using np.read_csv() The problem wasn’t that it didn’t work (the csv intentionally had a minor quirk that demanded that one uses a non-default option) but the reaction to it not working, or, in a majority of cases, the inability to search (on Google) for “what function allow me to import a csv file into a table” was incredibly telling. From there, you can have a conversation about what they did and it often leads to a more honest portrait of their previous functions. Sometimes, that led to offering them a more suitable position.
If I were hiring a VP of engineering or CTO, my only goal in administering such a basic test would be to see whether they are offended and prepare to politely leave the interview. Not kidding here.
Sure, CTO duties can run a broad spectrum, depending on company size, etc. It could essentially mean "dev lead with a small team", all the way up to public company CTO with a good command of financials, managing budgets, large multi-level teams, etc. But, at either end and anywhere along that continuum, I would want to skip to probing higher-order thinking around architecture, strategy, people-management, etc.
So, if I've brought in people who I believe are adept at skills like these, I'd expect them to be confused and perhaps even insulted when I pull out a fizzbuzy integer summation problem.
100% of those that couldn't do it are not qualified candidates.
Some of those that could do it are good candidates.
When you bring in someone that has a nice looking CV and experience and recommendations and you ask them a very basic question and they can't respond at all, it is a safe disqualifier. It also doesn't take six hours.
Looks like you edited your comment. In any case, that's a non-statement, as some is quite different from nearly all (i.e. 90%).
OP stated nearly the inverse: that 90% of those who can do it are qualified candidates.
That is, your numbers are only valid if I accept the premise that I'm rejecting. I don't believe that 90% of people who can answer such a basic question are qualified.
It's a great 15 minute question that I find often correlates highly with the people that get offers.