Amen to that, AI sometimes doesn't even choose the right answer even when a probe of its internal state shows that it has it. It won't explain itself correctly either. We need to burn our hand in order to learn, and for there's no instruction manual for something so emergent.