And I can see a difference between someone that hasn't voted, and someone that voted a 3-stars in a 5-stars system.
Well, I would say it kind of is, but it kind of isn't.
For many things, most people don't bother to think about the difference between crap unless you force them to, so in a 4-star system 2 stars becomes the "mediocre" rating while 3 and 4 differentiates between the good ones. How many people really care about a grade difference between D and an F? Likewise, do people really spend that much time making sure their 1 and 2 star ratings form a consistent philosophy of relative crappiness?
If "forcing" people to make a choice about something that is supposed to be a subjective categorization to begin with is probably not helping anything. If you want to force a like/dislike you should get binary data and be done with it.
"Would you eat this? Yes/No."
That's easy to answer accurately. Everyone will agree on what you mean. If you haven't answered that means you don't have an opinion. Beyond that semantic ambiguity is impossible to avoid and gets worse the more numbers you add.