I have a friend who has four restaurants in Tokyo, and I've been several times there. If you keep attention to the restaurant reviews in Google Maps, Japanese people is very hard. They'd go like "The food is great, incredible service, surprising flavors, very good experience, best Spanish food I've had in a long time..." and then they go "...but I know they can do better and have space to improve. 2/5".
In Spain we would be like "Nice beer, they gave us some tapas. 5/5"
Gig economy corporations also have made this worse. You're scared to score your driver, your server or your hotel by less than a 5/5 or somebody might get punished or fired (and maybe they didn't even had a contract in the first place).
And this becomes a snake biting its own tail. Now I rarely go to a place with a score of less than a 4/5, and I'll score relatively high because I know I could influence votes out of what people consider "worthy to visit".
I have a simpler heuristic. I ignore all customer ratings entirely, and refuse to leave any.
Coincidentally, NPS is entirely based on this fact. 8+ is required for a "happy customer". 7 is given a little wiggle room to win them back and under 6 is a lost cause.
Huh, could you give an example of it? I am new to the Netherlands and hadn’t realized this yet.
https://www.overstappen.nl/energie/vergelijken/
Under "Beste energievergelijker"
> Onze klanten beoordelen ons daarom met gemiddeld een 8,6.
If you scroll further, you can see their ratings for other services; none reach even an 8.
Compare that to Nerd Wallet's "Best Life Insurance Companies" (a segment that is nearly identical from all providers if you purchase term life).
https://www.nerdwallet.com/article/insurance/best-life-insur...
5/5 across the board for all parties. I get that it's a "best of" list, but you seriously mean to tell me all of these companies are exactly the same? The rest of Nerd Wallet's top pick lists are the same.
1-5. Ranging from pathetic to inadequate, in any case you didn't pass.
6. Adequate, the minimum needed to pass. A culturally important number as it expresses the "zesjescultuur", a phenomenon where somebody is intentionally doing the absolute bare minimum to not get into trouble.
7. More than adequate.
8. Good. Even if near-perfect, few Dutch people would rate above this number because we don't want you to get all cocky.
9. Excellent. In service, only awarded when you did something completely unexpected or memorable.
10. Perfect, flawless.
This is because 1 is the highest grade (mark) in school, and 5 is failing, as is the even-worse 6.
And even so informed, or accustomed to 1-5 scale used by everyone else, Germans are more likely than Americans to give a 4 instead of a 5 to something that they were happy with, but was not the best they’d ever experienced.
I also get the impression that American grades are like this:
- A is good. You made an effort.
- A [however modified with +/++]: great job!
- C is terrible. You skipped all classes or fell into a coma.
The last AirBnB I rented was a five-star house where the host had installed one of those Nest smoke detectors with the motion-sensing night light directly above the bed. He still got a five-star rating, but I went well out of my way to mention the night light in my review.
There was also a massive foam rubber pad on the mattress that made the first night a rough one... but again, some people who are wrong like those, and again, it was easy enough to remove. Or at least, it would've been if my partner hadn't already been asleep on it when I came to bed. The resulting tossing and turning was what made the motion-sensing night light a problem.
Basically I didn't want to go full Karen on the landlord because everything else about the house was comfortable and thoughtful. I felt that a four-star rating would have been projecting a combination of my personal preferences and comically-bad luck on his other potential renters.
It's a complete farce.
About skewed ratings:
0) there is no objective point of reference, every rating is about the reviewer more than the place.
1) google knows about it. they have weighed rating of some sort so 2/5 rating is not going to kill.
2) people in some cultures can give low rating without explaining what they didn't like. either it is complicated to explain what it is or it may be misinterpreted or not publicly acceptable.
The non-industry people reviewing Zelda have 4 options, 7/10, 8/10, 9/10, 10/10.
A 7/10 means the game was bad and you had a hard time finishing it, and won't be playing again.
8/10 means the game was also bad, but you had fun for a few minutes/hours.
9/10 means the game meet minimum expectations.
10/10 means you enjoyed the game, but there were countless flaws that took away from enjoying the game.
10/10 Greatest game of all time means, the game was above average.
Now, this same system applied outside Nintendo's curve, you knock these numbers down 2.
Its incredibly hard to figure out if a Nintendo game is good, if you treat them like other companies. Even Nintendo believes this and will yank early access/ads/etc... from gaming websites if you don't comply with the Nintendo curve.
They spend a lot of time on development, don't rush things, the trailers are honest about gameplay and what to expect.
I can understand it's not your cup of tea or arguments that their online service is shit (it is) but reviewing a video game in isolation, many of them seem to be good games objectively. Games that don't appeal to the average CoD fan sure, but that doesn't mean they are bad games.
Imagine you got rid of the Zelda skins and Zelda name, released it on Xbox. What would it get then?
Don't get me wrong, I religiously play all the zelda's and find ways to enjoy them. I am under Nintendo's spell, but I also know these are the corporate mascots I grew up with. Nintendo markets to children, we basically need therapy if we want to be free from their grip.
I do think it’s worth noting how many of the ideas from BotW have been copied in the past 6 years in other games.
I guess my point is, I can understand the 10/10s but I also find it entirely reasonable to view it closer to a 7 or 8.
We’re also not thinking about things in a vacuum in this discussion. Is the Xbox game released as if BotW never happened? Nintendo certainly uses its brand to buy marketing influence but importantly, I don’t think the company is pissed when games fail.
Skyward Sword was a good example of that actually. It’s failure literally led to BotW. Nintendo got off their laurels and reinvented their “tried and true” formula and now it’s paying off.
I am closer your position for games like 2D Mario. Zero innovation and I’ll be eagerly watching the reviews for the new game later this year.
You're right, but to be fair sort of 'reskin and re-rate' test would decimate the reviews of most sequels, remakes, adaptations, etc. Not just in video games, but movies, books, etc. Would Peter Jackson's The Lord of the Rings trilogy have been half as culturally relevant if it weren't standing on the shoulders of a very popular series of books?
It's important to review a game as it exists, and not a game that you imagined in your head that would be better.
> Imagine you got rid of the Zelda skins and Zelda name, released it on Xbox. What would it get then?
Well, in the case of Genshin Impact: lots and lots of money.
Seriously, though. If Breath of the Wild or Tears of the Kingdom had been released by a company other than Nintendo, people would be singing its praises to the high heavens. You'd see a lot of "Why can't Nintendo come up with this" type articles about it.
Did you mean the industry people? Plenty of user reviews are 1/10. It's the industry guys that are always rating stuff 7+
A couple examples: Lord of the Rings Gollum (a recent, high-profile, but poorly received game) has 36 critic reviews on metacritic for an average of 39, Forspoken (another similarly high-profile and poorly-received game) has 7 for an average of 63, and Tears of the Kingdom has 139 for an average of 96. Someone with more free time and motivation could pull a better spread of data to really show how the two data points relate, but I'd be shocked if there weren't a strong correlation between score and review count almost across the board.
This is of course Goodheart's Law in action but I don't have any ideas for fixing the problem.
It'd be definitely interesting to see a book review site like this. In the meantime, for math books, I trust curated rankings by MAA: https://www.maa.org/press/maa-reviews
As for the grandparent comment: recency bias, as pointed out by another commenter is a thing, as is the tediousness of doing a bunch of pairwise decisions. I think a happier medium is to have everyone fill in tier charts (with variable number of tiers) and build the pairwise rankings from that.
I've always been so for this in situations where the users also have a database of their own history on the site, like Goodreads or Boardgamegeek.com, for example. Just throw up modals every once and a while comparing two books that you know the user knows, and make it easy to opt out of. I probably wouldn't opt out, because it would appeal to my ego and feel like a game.
> Humans just aren't very good at ranking things on a normal distribution, so you invariably end up with every item (books in this case) being ranked somewhere in the 3.5-4.5 range (since Goodreads is out of 5). For IMDB the rankings all hover around 8ish. When in reality the average book should have a 2.5. If you don't rate like this then you just end up with garbage.
Just normalize the users against themselves. I wouldn't be giving a 4-star review, I'd be giving a pessimizer(4-star review).
1 - I hated it
2 - It was okay
3 - I liked it
4 - It was great
5 - I loved it
I thought it was helpful for framing what a score would be but I haven’t seen this in the app or anywhere for a long time. At least it avoids the problem of Amazon where a review is like “Best book I’ve ever read but the copy I received had a rip on the dust jacket - 1/5”.
Learned through experience to skip anything below a 4 or very high 3 now. Probably missing some interesting stuff but I don’t read that often anyway.
* Recommend this thing
* Neutral opinion
* Do not recommend this thing
Not sure how much "neutral" would be used: kind of like thumbs-up/down or up-down-vote mechanic: if you're neutral you may simply not vote at all.
But that also amplifies a problem that exists with any rating: recency bias, or any sort of time bias.
I change over time. An album that I loved when I was 15 could get a 1/5 from me today. Or a book that I hated in my teens, may now be my favorite. It's complicated.
Maybe also allow ratings to be a list over time rather than one, single data point. That'll make calculating an average more involved, but also more realistic.
I can't wait to compare The Pickwick Papers against The Unfolding of Language.
Constraining pairings within their genres might suffice, if possible. There are several standard ontology for books, and it's not a trivial problem.
You might as well ask the question: Who is the better Tennis player? Roger Federer or Garry Kasparov?
A system would need to draw some kind of boundary conditions so you are comparing things that have more in common than "printed on paper and bound".
There's the problem there – assuming human input makes up an important part of the rankings.
With eg movies a much greater investment is required to tell if it's terrible.
IMO the real problem with something as subjective as books or movies is that even completely honest, well-reasoned reviews are going to be all over the map. My review of a Pride and Prejudice movie is going to be maybe 3/10, but my wife would give it 7/10, while we have the opposite reactions to something like The Hunt for Red October.
I don't care about reviews from experts or the unwashed masses. I don't even really care about reviews that much -- I'm more interesting in ratings from people who like the same kind of stuff I do.
And for most people, it seems like the semantics of rating scales don't align with a normal distribution about the 50% mark, but something more like letter grades where 70% is decent/mediocre, while 50% is a failure (in the US).
IMO this makes at least as much sense as centering around 50%; if you only score 50% of the points on a test I don't think you're competent in a subject, and if your book only meets 50% of the criteria for a great book then it might be a bad book. But again, it's all arbitrary and only makes sense in the context of the particular reviewer or aggregate community.
I'm not sure it implies that. I think that if:
1. Book quality follows a normal distribution.
and
2. People are capable of accurately assessing the quality of books relative to other books.
then you'd end up with an average rating of 2.5.
It doesn't matter if people are disproportionately reading the better books and rating them highly because an increasing number of reviews on a book would only give you more confidence in its rating, not change the rating of the book relative to other books.
To illustrate this we can imagine a world with only 100 books in it and only 100 people in it.
Let's assume that a single person from that group reads all 100 books and per the two assumptions above correctly rates them on a normal distribution.
Then the other 99 people read only a single book which the first person recommends as a 5. And per the two assumptions above they all also rate it as a 5.
Now there are 99 books with only 1 rating normally distributed between 1 and 5 and there is 1 book with 100 ratings - all 5s.
Does this change the average rating of books on the platform?
Nope the average rating is still 2.5. In fact the one book could have a million ratings or a billion 5 star ratings and the average rating of books on the platform would still be a 2.5.
What we see on sites like goodreads is that either book quality doesn't follow a normal distribution and/or people are not capable of accurately assessing the quality of books relative to other books.
Terrible books likely have fewer readers, thus fewer and likely zero people would post a review.
When you go to rate a book the system asks you if it was better or worse (maybe same) then a book of you previously read closest in rating or some hidden matchmaking rating (MMR).
I think having books having both a personal and global score would be nice as well.
I don't think it would be possible for someone to change their rating but that may be a feature.
I also don't think this solves recommendation though it may be useful as a feature.
It should definitely be possible. Take films. I had a friend who would go see a film. On the day he saw it: "OMG THE BEST THING EVER". Next Week: "Yeah, that was a good film". A month later: "Yeah it was okay I guess". Six months later: "A bit ordinary really"
In 1990 if you'd asked me to rank the Alien films, I would have put ALIENS first, and ALIEN second. These days, I reverse those rankings.
People's tastes change over time. It would be inaccurate to "fix" a rating in stone.
People change their minds all the time (and it's a good thing), I agree.
While not impossible I don't think it would be particularly scalable.
Anecdotally this feels correct, the more common user rating distributions I see are an inverted bell curve - lots of 5's and 1's, not too many 3's.
But as far as the same problem that plagues every other review system goes, I would say the paid/fake reviews are the far bigger problem.
I played around with a tool for sorting through my personal photos based on that idea.
It gets interesting when you start thinking of how to optimally choose pairs to compare, to maximise signal and minimize redundant comparisons. This becomes more important as the size of the set of objects you are comparing becomes large.
It wouldn't be very helpful to read a review as to why someone thought Fundamental Calculus was better than Moby Dick. Without the star rating system, you lose the ability to filter on the reviews from people who think the product is bad.
Because of this all the review systems like IMDB are flawed by design.
Steam reviews are the most trustworthy for me. While they are not perfect, the binary choice normalizes these scoring differences, and can be read as a "probability that you will enjoy X if your are interested in this sort of thing" which is the best one can hope for.
And it is still useful. For example, from my perspective I can quickly judge a movie from an imdb score.
Anything less than 6.1 is bad. 6.1-6.3, ok, 6.3-6.8 good, 6.8+ excellent.
Apart from movies starring star trek characters, add one to all those scores.
Goodreads is similar but tighter, anything less than 3.9 is a red flag, over 4.2 and you start getting into excellent.
Though in my opinion steam seems to have got it done best with "overwhelminly positive", "very positive", "positive", "mixed", "mostly negative", etc. but occasionally gets review bombed.
We've done exactly this with Flickchart - https://www.flickchart.com - presenting the user with two movies at a time and letting them choose what, in their opinion, is the better film. We then aggregate everyone's rankings to generate a combined chart of the best films of all-time that's always in flux and changing.
We take into account each user's total number of rankings (refinement), and the total number of movies they've ranked (breadth of knowledge). We also use time as a factor - given that something that just came out has had less time to be "an all-timer" than something that has existed in the world longer and had the time to be experienced by more people.
We created this way back in 2009 and are currently building the next-generation of it on a modern web stack. This new iteration is also more modularized so we can expand more easily from just movies to other domains (books, games, music, TV, etc.).
Billy Olson’s sitting here and had the idea of putting some of these next to pictures of farm animals and have people vote on who’s hotter. Yea, it’s on. I’m not gonna do the farm animals but I like the idea of comparing two people together. It gives the whole thing a very “Turing” feel since people’s ratings of the pictures will be more implicit than, say, choosing a number to represent each person’s hotness like they do on hotornot.com. The first thing we’re going to need is a lot of pictures.
That's definitely not the case. 8 is neither mean nor median for titles on imdb. If a film scores anywhere above 8.0 they are already in the top 250. Most titles average between 5 and 7.
Interestingly that seems reminiscient of data apps. What are your thoughts on their ranking philosophy?
Humans just aren't very good at ranking things on a normal distribution, so you invariably end up with every item (books in this case) being ranked somewhere in the 3.5-4.5 range (since Goodreads is out of 5). For IMDB the rankings all hover around 8ish. When in reality the average book should have a 2.5. If you don't rate like this then you just end up with garbage.
Just allowing a boolean rating (ala Rotten Tomatoes when aggregated) is much better, assuming you can get enough reviews for that system to actually work (probably > 30 is required for most applications).
I think "aggregated personal Elo" would be a fun way to rank things: I just give you two books that you've read and you tell me which is better. Do this loads of times and eventually you have a solid ranking of every book you've ever read. Aggregate everyone's rankings and you have a much more robust system then "please rate this book out of 5 stars".