> come to the correct conclusion,

What makes you so sure that this is the correct interpretation? Both things could be true at the same time: there could be selection bias at the same time as critics are perhaps out of touch and snobby.

You could backtest this theory: see if perhaps critics scores were at one time a bit closer to general audiences or not. I think you'd find that at one time good critics were a bit closer to general audience scores.

It's also possible for critics to have systematically different preferences than movie-goers that don't deserve to be described pejoratively as "out of touch" or "snobby." In fact, that's precisely what I would expect and hope to see from people who are professional critics. I wouldn't want expect a list of best restaurants from a food critic to be identical to the list of top restaurant chains by revenue (1: McDonalds, 2: Starbucks, etc.).

> I wouldn't want expect a list of best restaurants from a food critic to be identical to the list of top restaurant chains by revenue

That's a very good point that the role of the critic is to provide their own expertise and opinions on the overall subject. But the critic still has to be within the ballpark of popular tastes if the expectation is a mass audience that really trusts their recommendations, right?

Many Americans might listen to a food critic that said that he thought some pasta dish at some Italian restaurant was the best, even if pasta is not currently everybody's favorite food. But if he he recommended boiled snails or braised calf brains, many people might ignore his judgement because it's a bit too far out of range of what they're comfortable with. At a certain distance from popular opinion, a mass audience might no longer trust that critic. Just a thought.

> But the critic still has to be within the ballpark of popular tastes if the expectation is a mass audience that really trusts their recommendations, right?

Not really. A professional critic usually writes a longish review describing the item and why they liked or disliked it, as well as the conclusion. If they are good at description, that's useful to me even if I don't share their likes and dislikes. If they hate things I love, then I can look for their negative reviews as places to start. If they love things I hate, their seal of approval is a good sign for me to keep walking. I can trust their recommendations even if I don't follow them, if I think they are fair and honest and have consistent opinions. If their opinions blow with the wind, I probably won't trust their opinions, even if they agree with mine.

I found Anthony Bourdain trustworthy, but I do not like a lot of the dishes that he really did (no thanks to organ meat and blood sausage), so I wouldn't blindly follow his suggestions.

> But the critic still has to be within the ballpark of popular tastes if the expectation is a mass audience that really trusts their recommendations, right?

Well, no, because I'm pretty sure most film critics aren't trying to directly provide a "yes or no" recommendation to a general movie-going audience. I've read plenty of film criticism that was entertaining and informative despite me having very little shared taste with the writer. And I suspect if you asked film critics to straight up predict what the RottenTomatoes audience score for a film would be, the aggregate of film critics' predictions would tend to be closer to the RT audience score than the RT critic score itself.

McDonalds is much cheaper than a fancy restaurant, so it should come as no surprise that it has worse quality. But with movies, people are chosing the lower rated movies even though it doesn't save money.

> It's also possible for critics to have systematically different preferences than movie-goers that don't deserve to be described pejoratively as "out of touch" or "snobby."

When the person writing a review has "systematically different preferences" from the people they're writing for, "out of touch" is exactly the right description for that ("snobby" - not necessarily).

Or put another way, maybe you’re not the intended audience if you’re reading a film critic’s work only to decide if you might like a film and you repeatedly find that your tastes are very different than that critic’s.

Looking at the mechanisms alone I think its pretty unlikely that film critics have not shifted over time.

The standard film critic job in the 80s/90s was a local newspaper. That meant your audience was men/women of various ages and political beliefs who happened to live in the same area. If you were a successful critic you had to write something that appealed to all of them. This was actually a vital service that people used to determine if a movie was good.

Now the typical movie critic is online and has an audience which probably has a singular viewpoint. This isn't bad - but makes it unlikely the critics with "everyman" sensibilities would do well. People who consume an above average amount of film reviews will have their views represented more.

Chuck Klosterman (who was once a local newspaper movie reviewer and now writes to a certain type of audience online) has written about this.

Gotcha type interview questions with a single "trick" that you either get/have seen before or don't provide terrible signal imo.

The comment literally states that it's one of the questions they ask, which is the exact opposite of a gotcha question.

And the specific question can lead to a conversation, which may "reveal" the answer (obviously, I don't know whether this is the correct answer), but at the very least it reveals whether the interviewee thinks about selection bias, among other concerns.

> The comment literally states that it's one of the questions they ask, which is the exact opposite of a gotcha question.

Huh? "Gotcha question" doesn't imply it's a single question that's not part of a series of questions. A "gotcha question" is simply a question designed such that it's very easy to respond with something that sounds bad or is incorrect.

Is it really easy not to come up with the desired explanation here? It seems like the most obvious possible answer, that people tend to watch the types of movies that they like. Actually it seems to provide almost no information unless their hiring pool is mostly candidates who utterly lack common sense.

I recall reading that someone found fizzbuzz (with nothing fancy) to be a highly effective filter for programming interviews

Hiring pools tend to be filled with nonsense candidates

Calling that the "correct conclusion" is absurd. Whether or not a critic "wanted" to see a movie shouldn't really factor into the review at all. At the end of the day, the typical movie-goer is probably judging a film based off of an entirely different set of criteria (excitement / satisfaction) than a typical movie critic (overall artistic merit). Which isn't an exciting conclusion, but probably more correct.

I would personally rate many movies I've seen lower if you told me to judge it from the perspective of a movie critic as opposed to a general audience member (The Purge for example).

No, the correct conclusion is "I don't know". There is no real evidence one way or the other. Their unfounded speculation about human nature is just as plausible as yours. In the interview you can generate a hypothesis, not a conclusion.

This reminds me of the "why are manhole covers round" question, where people are judged by their ability to come up with clever wrong answers.

> audience typically only goes to see movies that they are likely to enjoy

That doesn't really explain the opposite phenomenon: movies that have higher critic rating than audience rating (e.g. movies like Mulan and Turning Red come to mind). IMHO, if you were to apply that rubric across all movies, the conclusion you should get is that critics and general audiences are simply different demographics and that it is normal for discrepancies to arise from sampling different demographics, not that general audiences are more biased towards what they like to watch. For example, parents wouldn't necessarily pick kids movies as their first choice of entertainment if age appropriate-ness wasn't a consideration (I think a lot of parents can relate to being sick of replaying Frozen, for example)

Then how do you explain that there are systematic differences for certain film genres? The article has a graphic on it. The differences are by no means uniform. Crowds love Action and Thriller, critiques History and Documentary. D'oh.

That is not to say there is no selection bias. But the population of critiques and film-going crowds are almost certainly different in education, lifestyle etc. That has certainly an effect on the observed discrepancies.

and whats the explanation for the opposite? when critics scores are much higher than audience scores?

I assume "out of touch" but almost any explanation I think of and ridicule could be seen as a strawman argument, so I'll wait for your response.

I could imagine the answer being:

People went to see these movies not because they wanted to per se but because critics said they were amazing. And because they went mostly due to critical reviews, anyone who didn't like it was even more likely than normal to be like "wait I feel tricked this was bad"

This might be a bit of a stretch, but it seems almost plausible.

Two more possibilities:

- A film may have been marketed as one particular type of movie via trailers (e.g. as a horror film) but was actually a different type of movie, often a subversion of the genre it was marketed as (e.g. ended up as social commentary). So audiences feel deceived and report that in their ratings.

- Similarly, there are movies that are "applause lights" where critics are obliged to give positive ratings due to the social significance of a film (e.g. made by a woman / featuring a woman in a lead role). But the movie might just not be that good, and audiences report this in their ratings.

> social significance of a film (e.g. made by a woman / featuring a woman in a lead role)

for a less divisive example: movies about the film or theatre industry get high marks and fit your example while the audience is left scratching their heads

Excellent point. Critics have often remarked that movies about Hollywood tend to get Oscars just because of the subject matter, which Academy members like.

Also, a less cynical example: some movies are about something new: you might see people who live in a developing country, e.g., Somali fishermen in Captain Philips, or in unusual occupations (e.g., WWF wrestler in The Wrestler). Most audiences just want to relax and not learn something new, but critics love these kinds of movies.

Not GP, but looking at the relative difference graph[1] two hypotheses occur to me:

1) Most of the categories that are over-represented are ones that individuals watch out of obligation, rather than desire. A large portion of people watch documentaries (and biographies/historical films) because they feel they "should", because they feel like they should learn something.

Same for drama - these often include Oscar-bait that many people watch exclusively because it's nominated, not because they're actually interested in the movie itself.

Likewise, animation is (by volume) mostly synonymous with children's entertainment, so there are going to be a lot parents forced to watch what their kids are watching.

All of these line up with OP's hypothesis of obligation over choice.

2) War and westerns being overrepresented, well... film critics historically have a reputation of being Men of a Certain Social Demographic, and this is likely reflected even in modern film critic culture.

[1] https://stephenfollows.com/wp-content/uploads/2021/03/Differ...

I used to be a film critic for the largest newspaper in a pretty big US city. Movie critics absolutely do not watch every movie. Not even close.

Fair point, it's categorically true that no critic watches every single movie released and I definitely overstated that. A brief search on Statista indicates that about 700 movies are released a year, the average American sees 2.3 movies a year (in theaters), and professional critics review over 200 movies a year, some even review 300.

I maintain that given this disparity (2.3 movies a year versus 200 a year) my point still stands, critics will tend to watch movies regardless of any predisposition towards them whereas an audience member will be much more discriminating about which 2.3 movies they will pay to see.

I used to be a better professional film critic, mostly festival movies.

movie critics do watch the best movies. eg on a festival with 100-250 movies they watch from 30 to 50 typically. from these 30-50 there are about 20 good movies, the best movies of the year. from these regular moviegoers watch 1-2. normal people do not watch the best but the worst with the biggest ad budget.

of course critics don't watch every trash movie which starts every week. from 5-8 filmstarts per week, they watch 1-3. these weeks more, around 3, because now we have the best weeks of the year, with last year's Cannes movies arriving. and then again late fall, with the best Oscar movies being pushed. but eg this year's Oscars had only 2 good movies overall.

for comparison: my letterboxd https://letterboxd.com/rurban/films/diary/

If your narrative was true, we would always see critics scores lower than the audience (assuming statistically significant number of critics reviewing a given movie). But we see so many counterexamples with radically different ratings: Spy kids, Ad Astra, Noah, King Kong (2005), Babe, The Blair Witch Project, Chicken Run, and thousands of others.

Not to mention a number of films who stopped the audience scores, decrying "review bombing". Some of these films were fine, while some were objectively bad (-cough- Ghostbusters reboot -cough-). I have a feeling (though it is unsubstantiated rumor) that there are payoffs in the critics circles for certain movies.

It's possible that those movies failed to advertise to the correct audience. If they pulled in people who thought they were getting 1 experience, but ended up with a different experience, the reviews are going to be lower than if they hit the right audience.

Critics, as noted, see everything and so they wouldn't be affected by bad advertising.

I've definitely been wrong, in both directions, about movies based on their advertising.

I think another acceptable explanation is that movies are marketed not to everyone but to certain people.

So say the latest horror flick that comes around is marketed to teens and 20-somethings. The over-30 crowd gets turned off by violence and gore. But that particular demographic loves it, and the more gore the more they like it.

There's a certain group of people that loved Human Centipede, e.g. But it got terrible reviews. Roger Ebert refused to award it stars, but gave it a few positive notes regardless.

And in it he even said:

    I have long attempted to take a generic approach. In
    other words, is a film true to its genre and does it 
    deliver what its audiences presumably expect? “The Human 
    Centipede” scores high on this scale. It is depraved and 
    disgusting enough to satisfy the most demanding midnight 
    movie fan. And it's not simply an exploitation film.

https://www.rogerebert.com/reviews/the-human-centipede-2010

On the other hand, Ebert followed the critical consensus and gave Freddy Got Fingered (my personal pick for funniest movie of all time) a zero star rating. This is a gross-out comedy that succeeds spectacularly at what it set out to do, but very few critics were willing to judge it by the standards of the genre. The audience scores are very polarized (currently 56% positive on RT), but critic's reviews are only at 11%.

How does that challenge the argument that movies are marketed to certain groups of people? Gross out movies don't typically get marketed to soccer moms.

> most professional critics have to review every single movie

This isn't true. Quite far from it, in fact. Manohla Dargis at the NY Times, for example, averages only about 2 reviews per week. There are many more releases than that. Peter Travers at Rolling Stone does even fewer. Same with Justin Chang at the LA Times. Etc, etc.

IMO, the explanation is going to be multifactorial. Your "correct conclusion" probably explains some of the gap as critics are still obligated to review the prestige releases (even if they aren't necessarily interested in seeing them). But surely also does the divergence between the tastes of professional movie watchers and Joe Moviegoer explain the gap.

What's interesting is that in over-indexing on one bias, you've probably introduced another into your own hiring process. It does seem as though picking people who are careful, independent thinkers could still yield good hires. In the end, if you get a good candidate, who cares? I'd rather be right for the wrong reasons than just being wrong.

A comment on the article was: """

I reviewed films for the whole of the nineties for the BBC and local press in Manchester rarely missing any new releases. With this luxury and dream job though comes a responsibility and after a while it became obvious that I was suffering from Movie fatigue.

It was easy to dismiss much of Hollywood’s fare as formulaic and worthless because we had seen too much and seen better. Some of the movies I panned were perfectly good films and many were extremely successful and popular with audiences, but for me they were dull, and despite fabulous production values left me cold. Mark Kermode (Film Critic BBC) was one of my contemporaries and he said: “You have to see everything so that when something comes along that is good you will really notice it.” Of course, this exacerbated the fatigue and often meant that we were too quick to judge, and after a while, very little appealed because we had set ourselves far too high a standard to be realistic about the film’s quality.

If you look at the reviews in The Guardian, you will see more bad reviews that good. You might argue that a bad review is much better copy than a good review so there might be an editorial decision in there. To a certain extent Movie critics can be a poor parameter for quality and an injudicious contribution to its success or failure.

After ten years of reviewing films, I knew it was time to stop. I wasn’t enjoying it and I was probably doing some disservice to the film industry. Less is so much more. Harry Stafford – The Manchester Film School

"""

And, perhaps Manohla Dargis and Peter Travers watch far more movies than they write reviews for.

Written another way...

You interview people, ask them their opinions, and hire those that agree with your opinion.

The most excellent way I've seen to hire a group of people biased toward a singular way of thinking.

Actually, both statements boil down to the same with little modifications: (a) being out of touch with the audience is same as (b) the audience typically only going to see movies that they are likely to enjoy. (a) is broader than (b), can subsume (b).

You really don't think film critics being different from the general population has anything do with it? They're trained to see things in movies that most people don't. I get that you want people to say "selection bias" but that's not even close to the only significant factor at play here. You have two populations, which is also a classic statistical phenomenon. You even get close to this in your discussion of selection bias—they watch a lot more movies, and are thus more experienced—but you don't seem to want to allow this, which seems odd.

Also - a non-trivial percentage of ratings on Rotten Tomatoes & iMDB (and all review sites) come from accounts that have reviewed more movies than there is time in a human life to watch.

The amount fake user reviews can skew a rating is not to be taken lightly - and IIUC - Rotten Tomatoes & iMDB might be filtering out some/a lot of spam - but definitely not even close to all of it.

The same is true for GoodReads - the amount of teenage booktubers that have reviewed 10k+ books is awfully suspicious...

"A very small minority of UX Researchers come to the correct conclusion, which is that most professional critics have to review every single movie"

Could you provide your sources? This seems like it could be very incorrect, or only partially correct. Keep in mind rottentomatoes is owned by movie studios so they have a vested interest in changing the formula for disregarding audience ratings. I think they already disregard ratings below a certain threshold and other weird shit.

I disagree completely with your conclusion and I offer the same level of evidence you did.

“I hire people who think exactly like I do”

When you have the right mental model, this is an appropriate approach.

Not every mental model is correct, either locally to a domain or globally in all cases.

1 < 2, in almost all cases except for small Z_{0,1} or similar type cases where mod functions modify the space to something exotic or comparator functions are defined far out of the typical. If you think there is a case for 2 < 1, and you aren't appealing to something exotic, you have the wrong approach.

Any mental model that assigns zero weight to the probability of being wrong is a wrong mental model.

That said, biases arising from endogeneity might have negative effects too. You can't conclude a parameter should have a different/zero sign just because of endogeneity, you have to go fix your model, and re-estimate the parameters.

> Any mental model that assigns zero weight to the probability of being wrong is a wrong mental model.

This is true when there is uncertainty. It also doesn't connect to "I only hire people who think like I do", which is the context of my response, so I don't follow your point.

Agreed on your other point that you need to instrument endogeneity. Another requirement is plausibility. Hookworm presence in Greenland should be uncorrelated to solar sunspots.

I find it distressing how many people miss the obvious selection bias at play here.

"The score, made up of audience ratings on a scale of 100, is calculated by taking the percentage of people who rated it at least 60 out of 100 (or 6 out 10) and multiplying it by 100%."

The critics score is calculated with a different formula. Do you discuss this with candidates?

Maybe research your interview questions a bit more before using them.

I absolutely do go over that, I also go over Metacritic and point out that this phenomenon applies to movies, video games, music, books, and almost all review systems that segregate professional reviews with customer reviews.

The article claims that the difference between audiences and critics has been increasing over the last two decades. If the difference was only because audiences self-selected while critics had to review everything, why would it be increasing over time?

If anything, I'd expect that the audience would nown have more of a hit-and-miss experience than 20 years ago - with the abundance of streaming services at your fingertips, it's much easier to binge watch movies just because, whereas, say, having to rent a VHS tape had to bea more conscious and committal decision.

No clue, one moonshot people live more in a bubble and narrow their film selection?

That's a clever way of looking at it. Did you come up with it yourself? If not, I wonder what you think it says about a person that they were exposed to this idea before they interviewed with you, versus after.

No I did not come up with it myself. It's a common pitfall in statistics known as selection bias [1].

My problem is that if I ask a researcher about selection bias, they have no problem defining it for me and even giving me a textbook example of it. But when I present to a researcher a real life scenario that is highly susceptible to selection bias, they will almost all fail to identify it. It's a problem where many people are able to compartmentalize knowledge but fail to actually make use of that knowledge in practice.

As a run a quant firm I see this among so many researchers across many disciplines. They fully understand a concept when you ask them directly, but they fail to put those concepts to use outside of a very direct and artificial setting.

[1] https://en.wikipedia.org/wiki/Selection_bias

Do you agree that without proper research backing it up, you can’t say for sure that this discrepancy is due to selection bias, even if on the surface it looks like a nice fit?

If you had said that you’re looking for candidates to raise it as a possible explanation, instead of seemingly firmly asserting that it is the one and only correct answer, you’d probably have less replies arguing with you here (or perhaps if you do have such research readily available, to have shared it).

No I don't agree with you. I think my comment is a reasonable statement of the situation. Is it a 100% correct statement that is absolutely categorically true? No, but most people who are not looking to be pedantic and win an Internet argument won't go looking to nitpick the details of my statement.

What most level headed people will do is understand that my statement is about how you can not reliably compare a group of people who are predisposed to enjoy a movie to a group of people who review almost every single movie that's released, regardless of their predisposition towards it. People are welcome to argue that and honestly I'd expect nothing less, but it's a basic statistical principle that I expect every single researcher I hire to be well familiar with and if they're not, then I make the choice to not work with them.

>you’d probably have less replies arguing with you here...

There is also a selection bias among the people who comment on HN and they too are not representative of how most of the community feels towards a topic. The people who want to nitpick the specific wording of my statement are certainly entitled to do so and will likely find something to nitpick regardless of how I framed my statement, but it's not a particularly interesting discussion to follow up on.

Sure, though I think the other implication from the difference stated is that critics have sampled randomly or more often and can therefore recognize derivative work where a target audience that shares a limited interest in film will almost uniformly perceive novelty. I think realizing something is a poor knock off is the basis of snobbery.. so it is strange not to accept that answer.

That's a very strong assumption to make, I could easily assume that most casual movie viewers watch the same derivative knock-off action/horror/comedy films over and over and over again without issue. You think Adam Sandler is the highest earning actor/producer because his audience perceives a great deal of depth novelty in his work?

The answer that requires the fewest assumptions and is consistent with almost all basic statistical analysis is that a group of people predisposed to like a movie will rate that movie higher than a group of people who watch movies regardless of their predisposition towards it.

If you're using "selection bias" to refer to the fact that these two groups are selecting movies using different, biased methods, doesn't that imply that there's some way of sampling that would not be biased? But no matter what you do, an average critic and an average moviegoer are likely to disagree on many films. I would expect the explanations behind this to be more of a function of differences between the groups, which explains 1. why critics will occasionally enjoy a film more than the general population, and 2. why this disagreement can drift over time. Both of which are not accounted for by your analysis. I'm definitely not saying you're wrong - quite the opposite, just that there are other interesting things to discover here.

IMO it seems obvious to me, but that might be because I took a statistics class in college.

(Not “obvious in retrospect”—I’ve had a number of personal discussions with people where we talk about the difference between critic & user scores on Metacritic.)

You also don't have any emotional or financial investment in the situation, though. ;)

I don’t understand what you’re trying to say.

As someone that enjoys traveling and testing food places, the same applies to food critics.

Most of the time when I fall again into the trap of following their advices about some place, I come out disappointed, usually getting some creative dishes that still leave me wishing to go eat elsewhere.

I have discovered so many nice places just by randomly crashing into them, where one could watch from outside people were enjoying being there, regardless of how many stars the food critic gave it.

Wouldn't movie critics tend to be people to like movies in general? What about critics that stick to their preferred genre? They might start out pre-disposed to liking movies which could wipe out the effect of your assertion completely. On one hand, I think there is a good chance you are at least partially correct, but declaring your offhand intuition about movie criticism industry to be the "correct" answer is at best hubris.

> Those researchers are typically the ones I hire.

But there's no evidence that demonstrates either possibility (or if the reality is a 3rd, 4th or 5th possibility).

Never thought of that but it makes a lot of sense!

Edit: worth noting that this also creates an implicit prediction about the extent to which the advertising is an accurate representation of the movie. This could go in the opposite direction and audience scores would be lower than the critical assessment.

I wouldn't consider either of these to be the "correct" solution. You don't need to posit anything at all about how film criticism works. Occam's Razor suggests (ROT13):

Vg'f whfg erterffvba gb gur zrna pbzovarq jvgu erfgevpgvba bs enatr. Vs lbh unir gjb pbeeryngrq inevnoyrf K naq L, naq lbh cvpx gur zvavzhz A fnzcyrf sebz K, cebonoyl gur pbeerfcbaqvat A fnzcyrf sebz L jvyy unir uvture crepragvyr enax. Lbh pna pbasvez guvf ol fvzhyngvba. Gura pbzovar jvgu erfgevpgvba bs enatr: vg'f vzcbffvoyr gb tb orybj 0% fpber, fb ybj pevgvp fpberf "cvyr hc" ng mreb. Gura jura lbh ybbx ng gur pbeerfcbaqvat nhqvrapr fpber, nal aba-mreb inyhrf ng nyy jvyy envfr gur zrna. Ol flzzrgel, jr fubhyq rkcrpg gur zbivrf jvgu uvturfg pevgvp fpberf gb unir ybjre nhqvrapr fpberf, naq ivpr irefn va obgu pnfrf sbe ybj/uvtu nhqvrapr fpber.

Off topic question: Is there anyone here who reads raw ROT13?

A couple years ago, I could manage reading and writing ROT13 somewhat decently. I can still do it after some warm ups first. I can also do arbitrary Caesar cyphers in my head, but not fast enough to impress anyone. Memorizing the alphabet backwards helps, as does being able to visualize the numeric value of each letter. Embarrassingly, I'm actually rather slow at the simple mental arithmetic part.