I think this is a result of going along with what humans are good at rating. Given a reponse to a prompt, I can near-instantly rate whether or not I find it helpful, and whether or not I think it's ethical. "Devil is in the details" and the tails may come apart[0], but I imagine that in most cases, me, you and other raters would be broadly in agreement about helpful/ethical scores, and able to make those determinations very fast.

Not so with accuracy. I mean, this very reply is me giving you Human Feedback that I find your reply not accurate enough. You or someone else who replies to this comment will likely be doing the same to me. Or generally, give me a random AI response, and unless I'm the subject matter expert on that very topic, I'd have to spend inordinate amount of time researching it to give a reasonable determination. That's not feasible at all. Especially not on general-interest topics, where everyone has an opinion, and if any "ground truth" exists, it's buried 10 kilometers deep under an ocean of SEO spam and content marketing horse manure.

(And no, highly technical or mathematical topics aren't much easier - half of the discussions here on HN are living proof that even a highly technical audience, presented with a highly technical topic, will still heavily disagree about accuracy of things.)

--

[0] - https://slatestarcodex.com/2018/09/25/the-tails-coming-apart...