This is perhaps a ridiculous idea but I'll post it anyways. There are a lot of features of a paper that might correlate with it's truthfulness. For example surface level stuff like what journal published it or the authors, what subject it's on, how controversial/interesting it is. Then the data itself.

Then get a large sample of papers which are known to be true or false (or at least were able to be replicated or not) and use some machine learning or statistical method to make decent predictions how likely new papers are to be true or false.

How are you going to prove that your system isn't pulling random ratings out of thin air? ;)