I think one of the most robust rating systems I've worked with is based on comparisons, i.e. "did you like this more or less than X". This is more computationally intensive, but it can be made to work even with intransitive and unreliable judgments.