There are similarities to the approach in this paper (though they trained a model from scratch): https://arxiv.org/pdf/2309.07062

How well would an LLM trained with a huge number of examples do on this test? Essentially with enough attention, Goodhart's law will take over.