That's an interesting idea but it would exclude many humans who can make correct predictions using their experience and intuition but can't justify them correctly. Those people are still very useful.

What you're describing is what we do at school. We can't assess understanding so we assess justification of answers as well as other things like ability to do X (we don't care if they understood or not, just be capable).