I wonder if backwards a backward predicting LLM might do better.

E.g. augment the data with "Certainly, the document matching your question is <document>" and then sample the backwards completion to get candidate questions.