Instead of asking customers to give numeric star ratings, perhaps a better option would be a short list of Yes / No / Unsure questions. Were there clean sheets on the bed? Were there any cockroaches? Did the appliances work? Etc. Such questions are more objective and less prone to inflation.

This is probably a much better way to do it. One of the food delivery company does it this way (I think it's grubhub/seamless). Three questions before you're given an option to review: 1. Was the food on time? 2. Was the order correct? 3. Was the food good? Only then is the person given the option to review.

The problem with star based rating systems is that they're INCREDIBLY subjective. I oftentimes read yelp reviews that are 3 star reviews where the person raves about the restaurant and then complains about a minor issue. I have a similar issue with app store ratings where I've had a rash of people rate one of my apps 1 star with the review text reading "I just downloaded this, I'll raise my review to 5 stars if love this app".

Also a lot of times on Amazon the low reviews are reflective of an issue with that particular sale instead of the item itself (ie damaged while shipping, etc).

Another annoyance is amazon not allowing rating separate editions of something. For example a bad kindle version resulted in bad ratings for Dune a few years ago. Sometimes certain versions of movies are bad while the movie in general is great.

Booking.com does it, btw. And I strongly prefer it.

Surprised there is only one mention of Booking.com here. It is an indispensable tool for booking my vacations. The reviews are very useful and a lot more accurate than things like yelp. I have glanced at other websites, but nothing gives me the reassurance of the legitimacy of the reviews that booking.com does.

Agree - it really helps you identify which parts of the experience are problematic. If the location is 8.7 where everything else is 9.5+, you know you'll be walking a distance to get anywhere useful. If the location is 9.5 while cleanliness is 7.8, you can consider the trade-off to be in the thick of things.

So do low end hostel sites, funnily enough. Arguably they need it more to distinguish between the places that are popular because they're clean, good value and quiet and the places that are popular because they have a very good (and loud) bar and a vendor who's very good at persuading people to go on his tours

They also require more time, and most people don't bother to leave reviews anyway.

But if you're going to review, a few extra seconds shouldn't matter

It's how the "official" reviews from Marriott/Hilton/etc all work

1. Require each person to answer one specific question per listing. Yes/no/almost

2. Ask each person a different objective question, intentionally filling in the gaps.

3. After a few people, you have an objective, relatively thorough view of the house, but you've not burdened any of your users with more than one question per stay.

The new review flow on Airbnb asks for more details about specific aspects of a listing.

Is anyone trying to solve this right now? It seems like a really interesting problem that would be really useful to solve.

It isn't just online, and it isn't just a technology problem. I recently bought a car and the manufacturer sent me a survey about the process. The salesman made sure, numerous times, to let me know that I'd be getting said survey and that anything less than a 10 meant he failed and that there would be negative consequences for him. Since I was mostly happy with the process, I gave him all 10s, even though one or two areas I would have rated a little lower. He got his good review and it didn't really bother me, but the car company lost out on some potentially useful feedback.

Some of the problems - like different people having a tendency to rate on different scales, ratings coming from different factors, etc - seem pretty amenable to a ML approach to make reviews a lot more useful, though. I wonder if anyone outside of a handful of companies (Amazon, Yelp, AirBnB, Google, etc) has enough data to make it useful, though?

Many services have this issue, that perfection is the minimum requirement. I don't understand why, given this, we don't just go to a thumbs up / thumbs down rating system. We have one option that means "fine" and four (or nine) that mean "terrible, and a risk to your business".

I know I personally gave plenty of three star reviews on Lyft before I learned better, meaning them as "no complaints". Who knows what happened to those drivers as a result. Why even invite the misunderstanding?

This could be a cultural design thing. A lot of these apps (Airbnb, Lyft, etc) are from the US, where getting full marks in school is a frequent occurence if you study reasonably well.

In France for instance, it's very rare for a student to go above 15/20 or so at the college level (I graduated in the top 10% of my class with a 12/20 average; the top ranked student, whom I was friends with, had a 14/20 average).

But when I graded homework as a graduate student for an american university, the professor told me I was too harsh, and after adjusting my criteria to match what she wanted, I found myself giving many 100/100.

The US is also where you insult service staff and deprive them of income by giving them a gratuity which would be considered generous thanks in most tipping cultures. But at least that's baked into one's understanding of the price.

When it comes to ratings used by others to evaluate, no system is more customer and vendor-hostile than one where everything that meets a minimum threshold for adequacy should be rated as perfect.

It's silly that they have a system that pressures you into lying.

In a lot of ways, yes. On the other hand, it seems like a hard problem to solve - how do you get honest feedback in situations like this, where you want to use it to both improve things overall and judge your employees/contractors? Maybe you can't.

I guess, at least a small step would be to make it so values less than the highest rating don't count against you so negatively - it should be ok to be a 3/5 or 7/10.

I wouldn't even believe the salesman that anything other than 10 will get them into trouble. I'd just refuse filling it out.

Has anyone ever tried

1. normalize reviews to make a 4 star from someone who always gives 5 stars worth less than a 3 star from someone who always gives 2 stars; and

2. make reviews secret?

I've wondered this myself. Someone with a dozen 5-star reviews is essentially providing no information and should be removed from the equation.

Reviews are a problem, but not for novels.

Perception of books and concepts covered within are entirely subjective.

However, when it comes to service or product quality, you can be almost entirely objective in its value judgment.

>Reviews are a problem, but not for novels.

Hmm. I'm willing to accept that the average reader is a more sophisticated and thoughtful consumer than for some other product categories. And there may be less overt tribalism and fanboyism in book reviews.

But, in my experience, there's still a lot of people who will almost unthinkingly give 5 stars to a favorite author or genre. (And I know some of those people who are pretty uncritical about SF books or whatever.)

Goodreads reviews are still pretty inflated. Anything around or under 3.5 or so probably has some big flaws.

I have found no correlation between the Goodreads score and my own perception of the book.

Again, novels and products are entirely different in that you're hopefully shopping for a product by its specs. You can't 'research' novels beforehand (unless you count as reading other novels by the same author as research) and they don't have any specs to go by.

> However, when it comes to service or product quality, you can be almost entirely objective in its value judgment.

No, all assessments of value are subjective (and for most services involve subjective weighting of multiple different subjective quality factors.) And the mapping from an internal value judgement to a star rating is something which is a separate subjective evaluation, and one on which research has shown substantial cultural variation along ethnic and other lines.

While that may be true for other people, I have no trouble giving a purchased object a rating for its objective value (but I do excessive research on buying options beforehand). Even if the judgment is relative to other products, it can still be objective.

The rating may be inaccurate because you can only measure in full stars, but it'll still be objective and the inaccuracy is the fault of the rating system, not me.

Objective (adjective):

> (of a person or their judgment) not influenced by personal feelings or opinions in considering and representing facts.

This doesn't seem to be as big of a problem in Japan.

On the popular restaurant review site Tabelog, a 3.6 (out of 5) is a very good ranking; there's more 3-star Michelin restaurants in Tokyo than restaurants above 4.5.

https://tabelog.com/en

How about making you rank the places you've stayed at? You could do it by subset if there's a lot.

Or normalize the ranges between min/avg/max of the ratings given by each individual user to min/middle/max of the scale before merging the number into the total.

If you do it secretly like shadowbanning and introduce it softly by slowly blending between conventional score and normalized, it would drive a whole generation of SEO/social/share-economy experts delightfully mad. Think of all the free publicity!

Jokes aside, it might help to regard low ratings as what they are: a way to flag bad behavior, dead serious stuff. We should stop pretending that it is a game, then maybe solutions will be more likely to appear.

Fundamental challenge with star rating system is your perception of what counts as 5 star & my perception of what counts as 5 can be very different. It basically comes down to what your personal standards of something 5 star are. I am generally a very adjusting person, so minor glitches might not lead me to move from 5 to 2 star but someone could be more particular than me in which case his 5 star will move to 1 star at slightest issue.

It's a difficult problem. Survey response scales have probably been a matter of research and discussion before the web even existed.

There are particular instances (like in this case) where there are a lot of incentives for just about everyone to give 5 stars, 10 stars, or a positive and call it a day.

But, in general, there are issues with self-selection, bias toward the high end of the scale, ratings meaning different things for different types and classes of product, etc.