At this point, we are not there yet in terms of letting AI make business critical decisions based on its own outputs. Its meant to serve as a decision support system rather than a decision maker.

To minimize hallucinations, yes AI should be set up for deterministic behaviour (depending on your use case, for example, in recruiter, yes it should be deterministic so it produces the same evaluation for the same candidate every time). Secondly, having another AI check hallucination can be a good starting point, assigning scores and penalizing the first AI can also lead to more grounded responses.