Have you also tried the bigger models? The smaller models are good for assisted generation: https://huggingface.co/blog/assisted-generation

Those models of LaMini-Flan-T5 are trained to follow instructions and not to recognize the truth content. You could train a transformer like Ernie or Vega (which lead superglue) on such challenging factual data. But don't expect mathematical correct results only from the model. Therefore you have langchain with other APIs.