However, I think although this system might not be able to infer truth, it would at least be possible to infer "effort put into the post" or "well structured post" vs the obvious low effort things we see every day. Which might be enough for a better system.
Any suggestions for what you would train on to get a model which could rank more objectively than an upvote/downvote system can?
I think we should be vary of being bound by everything that happened in the 70s due to the differences in resources for solving the problem, although I'm not in any way saying this is easily solvable or even solvable in our time. I do think it's not provable that we can't achieve this at some point.
But really you have to define "good" "objective" and "structured" first. Or even "effort". Is the system good if it promotes high-effort well-structured calls for genocide?
We are simply asking for another racist AI or gorilla moment.
The only people who haven't given up on the approach of a massive symbolic reasoning database to work out what's true and false are Cyc. https://en.wikipedia.org/wiki/Cyc
You might be able to have a system which auto-flags known bullshit statements, but people will get outraged by that.