We've analysed product ratings in the past and found that the online ratings tend to have primarily bimodal distributions. People who rate online welf select and either love something​ and give very good ratings or they dislike and give poor ratings. Unlike surveys where people have to answer questions, only people with extreme reactions tend to answer these online rating questions and you don't get any meaningful distributions. So the only meaningful way to model say drivers from comments vs. ratings is to treat the ratings as binary and do a logistic type model.

Ultimately the key reasons to have these ratings in the first place is not to let people express their feelings, it is to get data for the recommendation modeling to predict better. Perhaps showing a simple like/don't like increases the number of people rating something which gives you more training data and thus better business outcome from your model without compromising on midel quality since you had to anyway model it bi-modally before.

I'm far from an expert on statistics, but wouldn't the fact that star ratings tend to fall on a bimodal distribution just mean that the ratings that don't fit the bimodal distribution have a higher signal/noise ratio than the ones that do? I'd think you could use a 2-4 rating as an indicator of a thoughtful reviewer. Additionally, it seems like you could also use the fact that a particular reviewer doesn't fit the overall bimodal distribution as a sign that this reviewer's reviews are similarly more thought out.

I've built recommender systems at Etsy, among other places, and you get much better results if you make the data yes/no first. Shockingly MUCH better results. You'd think it would help to take into account, say how _many_ items of a shop a user likes, rather than just the fact that they liked some item, but it doesn't – it actively hurts. The only meaningful piece of information, it turns out is does this person like this thing or not.

That doesn't really surprise me. A lot of people imagine that highly granular, multi-dimensional rating systems must be best and they're often not. Of course, it depends a bit on what you're trying to distinguish but then it may not really matter to distinguish among the 75% that people are genuinely happy with.

The fact that a reviewer uses 2 and 4 ratings might not be too meanngful. One reviewer might use 2s and 4s the way another uses 1s and 5s.

There may be reviewers who do not generally fit the bimodal distribution, but if they are the minority, why design your whole rating system around them? Better to optimize your system for the way most of your users behave.

yes scaled ratings do tend to have subjectivity which in practice we normalize by databasing surveys over time for categories x markets.

The reason online ratings tend to be bimodal is because the respondents are self selecting sample rather than randomized. Since product ratings are not typically compulsory, the extreme likers/dis-likers tend to fill the survey question.

When implemented as compulsory, it drives poor user experience and having to fill a survey to get something anyway introduces a bias (just fill something to get it over with). So no easy answer.

This is also the reason traditional survey firms haven't really gone out of business even with the wealth of online data from Facebook, Twitter and product reviews available, though better/easier access to actual behavioral signals are certainly replacing surveys in some areas.

I have no doubt you are right about the people who rate online (you definitely see it in restaurant ratings and comment sections) but when I rate movies on Netflix I very, very rarely ever rate something a 1 or a 5. I would imagine that 2/3 of what I rate has been a three (so average content for me) and that it takes something special to go up or down from there.

Just curious if I am an outlier with how I do my ratings.

I'm the same way.

1 - Terrible. Good Lord, who thought making this was a good idea?

2 - Poor. Would not watch again

3 - Fair. Neither for or against. Unlikely to watch again, but I don't grudge the time spent.

4 - Good. I enjoyed it. I would probably watch again.

5 - Exceptional. Probably in my top 10 now, and foresee enjoying watching for years to come.

3 and 4 are used a lot. 2 is used occasionally. 1 and 5 are rare.

I prefer the Goodreads rating system which compresses 1 and 2:

1 - Did not like it

2 - It was ok

3 - Liked it

4 - Really liked it

5 - It was amazing

The reason being is that most books won't end up being 1 or 2, and there really isn't much difference between them. But it sucks giving 2 stars to something that wasn't bad. Perhaps emoji arranged in a non-star-like pattern would be the best.

Interesting. I pretty much only rate 1s and 5s, hoping that this will influence what content is recommended to me.

I've heard this a lot. When I look at reviews anywhere where there is a star rating and text review the only useful reviews tend to be 2-4 star reviews. 1 and 5 star reviews seem to have a tendency to be outliers or people who are really upset about something that most people won't care about. For example 1 star restaurant reviews that waited for someone to take their order and left because it took too long. I wonder if that could be applied to weighing the ratings accordingly. I have no data to test this unfortunately.

> They are diluting the meaning of quality and instead are opting for a saccharine "enjoyment" factor.

They are just trying to increase number of views for a poor quality library. This will mask how bad most of the content is and increase their numbers.

Like many people I've had a Netflix account for a long time - I can't recall the last thing I watched on their streaming service. It has probably been four months since I've used it to watch a movie for example. Their library is increasingly complete trash outside of a few original content programs. I'd cancel it if it weren't so cheap, mostly I keep the subscription as an entertainment fall back (in the category of: well, maybe I'll watch something on Netflix, oh, nope). I suspect I'll just ditch Netflix in the near future and be done with it, as I'm pretty happy with Prime as essentially a freebie to my Amazon shopping.

Or edge cases like me will get that little boost to just give up on Netflix. I already use them less than other streaming services, and I could always just turn it on and off again.

Honestly one of the only reasons I keep it is because I have so many movies that I've rated and didn't want that time investment to go to waste. I guess now that isn't going to be a factor.

This!

My dream is to build an open source app that gathers all these personal ratings (from Netflix to Pandora to GoodReads to restaurants) and puts it back on where it belongs: the users hand!!

Unfortunately finding the time to work on this will be tricky..

Anyone knows if that's a thing already?

I've rated thousands of movies and shows on Netflix, not just what I streamed or received from Netflix on disc but anything I've seen anywhere that I can recall and came across on their site. Having rated something is also a reminder that I've seen it. I didn't want to lose all that so I downloaded the pages [0]. I'll figure out how to scrape them and put them in a better format some other time.

[0] https://www.reddit.com/r/netflix/comments/5zzc4g/batch_downl...

If you go to your ratings page and scroll down so they all load this worked for me (paste into console)

  ratings = [];
  jQuery('li.retableRow').each(function (i, row) { 
    ratings.push({
      id: jQuery(row).find('.title a').attr('href').replace('/title/', ''), 
      title: jQuery(row).find('.title a').text(), 
      rating: jQuery(row).find('.starbar .personal').length 
    }) 
  });

then

  JSON.stringify(ratings)

Or for CSV:

  ratings = '';
  jQuery('li.retableRow').each(function (i, row) { 
    ratings += 
      jQuery(row).find('.date').text() + ',' +
      jQuery(row).find('.title a').attr('href').replace('/title/', '') + ',' +
      '"' + jQuery(row).find('.title a').text().replace('"', '""') + '",' +
      jQuery(row).find('.starbar .personal').length  + "\n"; 
  });

then print out the ratings variable.

That's the streaming site page which has less information and other problems (follow the link).

I pulled mine from https://www.netflix.com/MoviesYouveSeen. I did notice a bunch that just said MOVIE but that was just a handful. My ratings page goes back to 2005 and I'm not subscribed to the DVD plan anymore.

Why on earth would you use Netflix as a rating service? First of all their library lacks a lot of great movies, or info about the movies they have. Netflix isn't IMDB.

If you include the DVD service they have practically everything. I've been rating movies I've watched in Netflix since 2005-ish. IMDB makes sense, but I chose Netflix and now I'm stuck with them. Data silos suck.

>If you include the DVD service they have practically everything.

Though quite a bit less than they used to. My observation is that more and more back catalog type of items are no longer available.

They're definitely not replenishing stocks of some DVD's, which I suppose makes sense given the trend. I would assume if they get a certain number of requests they probably do though?

Or is that my wishful thinking?

Maybe? The thing is that the incentives don't really align to make this a real priority of theirs and they've been pretty clear that shipping physical DVDs is not their long-term business model. So long as a lot of people don't start canceling their DVD service because of unavailable disks, they don't have real reason to replenish.

Of course, given that they pretty much put every other rental place out of business, your only choice in a lot of cases is just to buy a disk if you want to watch something that isn't available streaming.

Almost all of the ratings I did were back before they even had a streaming service and the library was great. Plus it was helpful to get good recommendations. Now it doesn't really matter, but all of those ratings are locked in.

I've looked before halfheartedly for a way to export them but now that they cut the API off it doesn't seem like there's a great way to do that.

This lock-in is a pain. A similar thing happens with music streaming where getting playlists between services is just awful.

That is one reason I've always preferred to control my music directly. Music in particular is something I can't stand to have discontinuity of playlists and collections with.

Spotify reordered my lists. It frustrates me just less than the chore of fixing them would, which is a special class of irritation.

I hear you. It's one of the few things that actually elicits a physical response me from... a kind of shiver of anxiety. I put time and effort and maybe even a little bit of myself into my music lists... it's personal!

That was definitely part of it or me too; the rating system has always been a draw. It took time to make it actually respond the way I needed it to, and now it's just... sigh

That used to be a big deal for me too. I had a little happy moment back when I rated my 1000th movie. Oh well.

What other services do you use? I paid for prime, so i've watched some stuff on prime video, but there's really not much on there. Netflix is pretty much my only option for most TV shows. Worth noting I'm in Canada.

If you enjoy Anime or Asian drama (asian drama being live action, not animated) Crunchyroll has a huge catalog. It fills the void that Netflix has between two big Netflix Originals.

Edit: Giving away one of my 48h Crunchyroll premium guest pass. No credit card required. I'm not working for Crunchyroll in any way, they give out guest pass monthly to premium users. First one to use this code wins: UWWJAQTDKYZ. Note that I will receive a notification telling me your username and perhaps your email address, if privacy is important to you.

I'm in the US here, so I'm not sure where out options differ. For me, I do (as a poster below noted) like anime so Crunchyroll is a good choice. Hulu is good for seasonal TV stuff, and then I actually use a smartDNS to get access to UK channels. It's probably a matter of taste though... I don't like most movies, and prefer longer-forms for telling stories.

>use a smartDNS

At that point, isn't just plain old piracy more convenient ?

Sadly, the simple answer is "yes".

I used to use a VPN to watch US Netflix, but they started to block known VPNs and I didn't feel like spending time on workarounds. On the other hand, downloading the latest episode of some TV series takes only a few mouse clicks. For sure easier than fighting geo-restrictions.

I still pay for Netflix, but doesn't really watch it much because of the tiny catalog available to me locally, I simply want to support them. Perhaps it will make me feel better if I would download any of their content. :)

>Sadly, the simple answer is "yes".

>I simply want to support them.

Nothing to be sad about. Also curious, why do you want to support them ? Being a market leader, Netflix is a unique position to influence some of these regressive practices. More support is clearly not helping; it's only making it worse by creating monopolies that exert even more negative influence.

The biggest problem with Netflix is that it is not able to obtain rights for a lot of movies or series so you don't have a full catalog. So series and movies are fragmented between providers or totally unavailable​.

Netflix is not much of a market leader until it contains at least 90% of the movies i want to watch, which can't happen without enough leverage from their part given by user support.

I do like some of their own content. I also like their general business model and I believe (well, I hope) that the stupid geo-restrictions are results of old-school licensing and not active business decisions made by Netflix.

But ofc, I wouldn't mind if they had some real competition, I don't want a monopoly.

>(well, I hope) that the stupid geo-restrictions are results of old-school licensing and not active business decisions made by Netflix.

Geo-restrictions have existed forever. What Netflix started was actively blocking entire ranges of IPs belonging to vpns and commercial ISPs (cloud providers like aws, dedicated servers like ovh etc.). This should fall under some corollary to net-neutrality where a service cannot discriminate between ISPs. Does such a corollary exist in countries that have some form of net neutrality (US, India, Netherlands etc.) ? Can someone with legal expertise comment ? From my standpoint, they can have geo-licensing, but should stick with some consistent way of showing content either based on billing info or IP. Blocking out vast portions of the internet because it doesn't suit them should clearly be some violation of laws like net neutrality.

There is a proposal in the EU that would prevent "unjustified geo-blocking" but I'm not holding my breath waiting for it to become law.

I've been so long out of that game, I wouldn't know where to begin, so not for me, no.

I have to say, I quit Netflix a few years ago after a long membership, due to security concerns (it was a nightmare at the time). Recently I started my membership up again, and I'm really disappointed with their library. Every movie I've wanted to watch has turned out to not be available for streaming. They've got some good shows, but I'm getting tired of super heroes and I don't always want to watch television, sometimes I want movies.

I'm on the brink of cancelling my membership as it is, this certainly won't help, though it's obviously the proverbial straw rather than a big deal in its own right.

Indeed. Note that Netflix has never offered a filter by rating, or the ability to hide already-watched or disliked content.

When I watch netflix, I use the stars as a measure of if I would enjoy something. I don't think most people use stars as a measure of "quality" because when they browse movies, they are looking for something they enjoy, not something that's high quality that they might not enjoy.

This seems like pure difficulty for me. When I think of thumbs up / down metrics I think in terms of "would I endorse this?"

A movie like "Suicide Squad" was (to me at least) highly enjoyable, though I thought it objectively had many problems with plot and delivery. I would probably give it 3 stars, which is relatively neutral for me (or leave it altogether unrated). If it were thumbs up/down, I would probably have to leave it unrated completely.

Maybe that's an improvement, but the more discussion I see around this, the more I realize the futility of single-axis metrics for things like this, until and unless you're asking a single question "Did you enjoy this movie" is a much different question from "is this a good movie" -- and I don't really know which one I've been answering on Netflix all these years past.

I think you are an anomalous user. I'm not sure why you're OK with rating 1-5 stars but not a thumbsup or down. I think you're just upset because they are changing.

I like giving weight to my vote.

3 stars = "not a bad movie, but wouldn't recommend"

4 starts = "decent movie, would recommend"

5 stars = "great movie, would definitely recommend"

A binary choice doesn't have any weight to it.

I don't think there's a need for weight since there are many reviewers. The weight is the percentage of people who thumbed it up.

How does weight have anything to do with amount of reviewers? Those are completely unrelated metrics.

The current suggested rating is influenced by your history of rating, with weight given to how highly you rated it. There are shows that are suggested at 4 stars for my wife but at 2 stars for me.

This. And based on my history of rating, Netflix had been pretty good at guessing if something will be 3+ stars. If they think it is 3 stars, I consider watching. If they think it is 4+, I tend to really enjoy the suggestion. Though, nearly every Netflix original is suggested at 5 stars, lol.

I agree. I even want a couple more increments between 2,3, and 4. There are a lot of movies I see that I think are better or worse than a 3 rating but don't justify a 2 or a 4. I feel there's a lot of room for nuance around the middle area.

Lets say a movie previously would have shown 3 out of 5 stars. Where does that fit in with the binary thumbs up/down? I think they are just doing this because recently some actors movies are being attacked with 1 star ratings for being politically vocal.

> I think you're just upset because they are changing.

This is actually rude of you, bordering on ridicule. He explained very clearly why he is dissatisfied.

I've had this argument with my (tending towards elderly) parents while introducing them to Netflix. They don't understand rating for enjoyment, and keep rating for "quality". They'll watch a show, absolutely hate it, but give it a 4 or 5 star rating because "it was well produced and the camera work was good, but it isn't the sort of thing I personally like to watch".

I'm worried about the thumbs up/down, as Netflix has increasingly recommended shows I do not like this year, and I don't think the thumbs solves the "rating for perceived quality" issue. I've been using Movielens for recommendations a lot instead, and it seems more accurate for me... and it has enough stats to show that on their site, most rating is not binary, much closer to a normal curve for average movies or a linear graph on really good movies. (It lets you see the distribution curve on individual movies, and other metrics.)

interesting. I always rated for enjoyment.

I thought that Neflix made it pretty clear that the ratings were used to choose which content to show me. So I should only rate based on what I'd actually like to see.

I think a lot of people look at the stars as a measure of enjoyment when trying to find something to watch, but when actually rating something, as a measure of quality.

I use the stars to signal quality as a means of determining if I'd enjoy it. 5 stars? This is likely to be high-quality and enjoyable. 3 stars? This is going to be lower quality and I may not enjoy it. 1 star? This is likely shit quality, and I'll regret playing—or it's a cult classic and all the haters don't get it.

They're optimizing the wrong metric.

They should be optimizing "number of people who pay for Netflix". Instead they are optimizing "number of hours spent watching Netflix".

Those two metrics are correlated, but definitely not fully.

If I spend hundreds of hours a month watching meaningless action movies and TV on Netflix, I can cancel my subscription and save myself hundreds of hours and $10. I've got lots of other ways of filling my time with endless pablum.

If I spend 2 hours watching a thought-provoking documentary, I say to myself "yup, that justified spending $10 on Netflix this month."

Accounting for your average person, who do you estimate is more likely to cancel: The person watching hundreds of hours of meaningless action movies and TV, or the person subscribing but rarely/never watching anything? By average I mean not prone to suddenly realizing they want/feel sufficiently motivated to spend those hundreds of hours on something else, or posting on hacker news.

Speaking anecdotally, the one (temporary) time I canceled was when I found myself in the paying-but-not-using category.

How do you know what metric Netflix are optimising?

They tell you so in TFA.

> Users would rate documentaries with 5 stars, and silly movies with just 3 stars, but still watch silly movies more often than those high-rated documentaries.

it's interesting because the more someone uses netflix, the less money netflix makes.

Not necessarily. I strongly suspect (but can't prove) that they sell their rating data back to the studios.

> The stars are not about "enjoyment" factor, they are about perceived quality.

For you. But are you representative of the whole or even a majority? I personally use it to reflect whether I enjoyed it or considered it a waste of time.

I give some of my favorites - like Hackers - 5 stars but would never consider it a "quality" movie. It's awful and silly but that's the fun of it.

Netflix have said for years that the star ratings they gather are not actually very good indicators for their recommendation engines compared to implicit signals such as - 'did the user watch the show / film to the end?' If they still believe this, it seems fairly irrelevant what they use.

> implicit signals such as - 'did the user watch the show / film to the end?'

I hope their version of that signal is better than what they're presenting in the UI. I have a few movies Netflix shows me as not finished, becaue I closed them just as the ending credits started.

Also, it seems incredibly delicate.

Upon finishing the final season of Dexter, I clicked on an episode in the first season to look up something. I jumped to the 18 minute mark and watched about 10 seconds and closed the tab.

Now Netflix asks me every time if I want to continue watching Dexter from where it now thinks I left off in season one even though I spent a whole year watching all of the seasons.

So I have the same concern. If they have a more sophisticated model, it certainly doesn't get reflected in the UI...

The UI... They can't even make an 'f' shortcut key in the Win10 app (app, not browser) to go fullscreen.

And they force 21:9 movies to be played back inside the 16:9 portion of a 21:9 ultrawide monitor, with massive black borders as a result where there should be none.

I'm seriously unimpressed with Netflix as a company, for what they do with all their resources. Sometimes the MVP just isn't good enough.

Shouldn't this quality rating be within a genre? Not all documentaries are 5-star, and not all silly movies are 3-stars.

How do you compare ice-cream with cheesecake?

Yeah, if they went with up/downs but allowed genre up/downs I think this would correct some anxiety about this change. "Yes, I still want to watch anime. No, the last 4 shows you recommended to me were not good."

I don't work in the recommendations side of the house, but the way I understand things is that nowadays your star ratings (and now up/down ratings) are primarily to learn about you, what you liked and didn't like and therefore what you might like to see in the future. They're not really about creating a Rotten Tomatoes or IMDB style database of trying to rank all movies on some sort of objective "regardless of what you like, this is a high quality movie" sort of scale.

> The stars are not about "enjoyment" factor, they are about perceived quality

I ignore ratings because of that. They don't mean anything and I am guilty.

On Good Reads, I have given what is deemed as a trashy piece of writing 5 stars. At the same time, I have given Dune 1 star because the story was garbage.

So what is this 5 star rating? Some people say it is the experience, some say it is the writing style, others say it is the story telling, or the plot.

Lets also face it, when it comes to films, the news papers have their own agendas and personal commitments to other agencies. My film could be absolutely garbage, but since I dined with the Editor a few times, It gets 4/5 stars and then, it gets 4/5 stars because the rest of the population is told how to feel.

I would even say that this whole system of rating based on Stars or Likes can simply be removed. If a film is good, you will know about it. If a film has been watched 1000000 times more this weekend than another, base it on that if anything.

When I rate things it's usually an "overall" rating, rather than weigh one dimension disproportionately. Which is why I would almost never grant a 5-star rating to something. But I'll dish out 2, 3 and some 4s regularly.

That is only you, I can't use that information. Like you might be taking into consideration "Special effects were 3 on their own". Whereas my Mum would never even consider that as a thing which influences the rating.

I understand that. It's not perfect and I don't think the assumption is that it was but more that overall people as a group will smooth those rough edges. In other words the majority will compensate for the minority who use the ratings system differently.

For all the faults of the old system, the new one is even worse. Like Pandora, I'll probably just never rate things or mostly just give them a down vote if they don't crack a threshold of goodness.

Which is fine, because the recommendation engine should be using your history, not someone else's. They gave it a 5, you have it a 1. They will see recommendations for similar movies and you won't.

Your thesis is basically, ratings mean absolutely nothing, and how good something is should solely be based on how popular it is?

The problem as I understand it is that the "rating space" is multidimensional (a piece can be funny with great plot but bad writing compensated for by a great lead and awesome special effects although the set design was inconsistent) but all of this has to be projected into a 1-dimensional system.

And the rub is that not everybody uses the same projection, which means the utility of one person's rating for another unknown their projection system (the weight they give to each sub-component of the overall rating) is completely unknown.

A much more succinct way of making the point. Thank you.

This is a very clarifying way to put it thank you.

Yes, the whole thing is meaningless but atleast views offer some glimpse. I would have given Barb Wire 6/5 as I was a teenager with a crush on Pamela. How can that relate to a rating of a film that you could use?

I agree with you. And hat tip for the nice analogy. Perhaps they should implement a system analogous to facebook's Reactions (e.g. "disliked", "thought provoking", "easy watching", etc) as a way to differentiate between cheap/easy and high quality releases.

But there's no difference in price between going to see a silly movie or a"good" movie.

If you are sophisticated enough to appreciate "foreign documentaries", you should have little trouble discovering them by another means than Netflix rating algorithm. The latter is for the laziest, couch-potatoest audience, and it makes sense to optimize for that.

Some friends of mine used to have a movie rating site which asked two questions:

1) How good was the movie? [1-5]

2) How likely are you to watch it again? [1-5]

I think it was too many questions to ask, but trying to separate out the enjoyment from the quality struck me as a good idea.

But Netflix doesn't care about the quality of a movie. They care about whether you would watch it or not.