The oracle is infalliable in the toy scenario. That was asserted as an axiom. To us who are god's eye observers of the toy universe, we already know that whatever the oracle says necessarily fits reality in that universe.

But I understand what you mean. Suppose I am Archimedes in there. I don't have access to the axiom, so I don't know if what the oracle told me is true until I can verify it myself. So verify I will. This is what is being done with AI-written proofs IRL. Humans may not be able to internalize the proofs, but we can internalize the programs which verify the proofs. So we do know the proofs are correct, or rather, as correct as the verifier programs.