> a problem of all averaging out to 3.5-4.5 stars

This is not a problem, it's sampling bias. If you went to restaurants randomly, then your average rating would be lower. But if you purposely select restaurants that you tend to like, then the average must be higher. Same for books, music, and everything on Amazon.

This is where Netflix ratings really shine. The Big Lebowski on my Netflix profile shows as five red stars. On my wife's, it's two red stars. Netflix guesses that I would really like the movie, while they don't think my wife would enjoy it much at all. And that's completely true. However, they give her a rating of five red stars for Masterchef, a show she actually loves. They give me four red stars, because I've rated other reality shows and other cooking shows lower in the past.

So if Yelp had that kind of thing, this person rated McDonald's highly, they'll probably like Burger King. This person liked Subway but hated McDonalds, so they'll probably like Jimmy John's but not Wendy's. Even more specifically, this person liked Wendy's spicy chicken sandwich, they'll probably like Subway's sriracha buffalo chicken sub.

That's the kind of investment and innovation that drivers user interaction.

I've gone to highly regarded famous restaurants that should have easily been 5 star.

The 2 and 3 star complaints? Trivial crap.

"Sure the food is great, but other restaurants also have great food, 3 stars."

"There is this other restaurant on <other side of the country> that has food as good at this restaurant, and my meal was $30 cheaper!" (for a $200 meal)

"I had to call ahead to get reservations, they couldn't just seat me, 2 stars"

Crap like that ends up with Michelin Star rated restaurants having equal "star" ratings on Yelp as the local Subway.

Well, stars should be a function of both expectations and service, so I don't think it's wrong for both a Michelin Star restaurant to be rated equal to a Subway.

They're probably measuring the wrong things. Maybe they should have something similar to the AirBnB / eBay ratings, which measure a lot of categories.

"Gave me extra cheese for free on my sub!" 5/5

If the reviews fail to provide an accurate assessment of the product's value to the potential consumer considering the purchase, then it's a problem. Sampling bias would be the root cause of that problem.

Something like a system that lets people say "This restaurant is better than that restaurant" or some other simple way to establish a partial ordering?