The “auto preview” is so unpleasant that I’ve essentially stopped using Netflix entirely. I’ve no doubt that their testing shows it increases engagement, but I find it so jarring and abrasive that I rarely open the app any more. I’m sure they still have content I want to watch, but I can’t stand the experience of trying to find it.
I would love to have more insight into their A/B engagement metrics. I’m sure they capture if immediate engagement goes up when a feature is enabled, and probably the same over time periods of days or weeks. Do they also capture if total app usage declines 75% over 6 months after a feature is enabled?
Netflix already seems to have no problem getting people to watch hours and hours of shows as a result of having high quality content. Why do they need to perform cheap tricks like auto-preview to bump up user stats, as others have pointed out, at the expense of user happiness/preference?
It doesn't help that their website is a laggy piece of shit with miserable discover-ability. But, you know. Ugh.
The website doesn't autoplay, but per Ntrails, it's "a laggy piece of shit with miserable discover-ability". I don't actually use the website so don't have an opinion there.
If you find a lower powered device such as the previous generation of Amazon Fire TV, it does not auto-preview or auto-run trailers at the top of the screen - instead you just get a static image of the show.
Now if only I could get it to do that on my PS4 :(
A/B testing won’t ever get you a Tesla from a horse.
Right. But if you were already selling Teslas, and along came a smooth-talking product designer with a dream to "improve" it by building a horse instead, A/B testing would make sure that change never saw the light of day.
A/B testing is not a way to get out of having to come up with good ideas yourself. It's a way to validate that your ideas are any good in the first place before betting the whole company on them.
A/B testing is most critical when evaluating big changes to a product, because those are the changes most likely to completely blow up the business. Otherwise it's left up to the opinion of the highest paid person in the room, and people are notoriously bad at guessing how customers will respond to change.
There’s something very powerful just talking to the user, feeling their pain and working with them to fix it until they say “this is awesome”.
How do you think the eyeball arose from unicellular eukaryotes?
You’re not going to get to a global optimum driven by human choice of what to test (only local optima at best) unless the human setting up the tests is some sort of sage.
For a trivial and well-known example, there is now no developmental or evolutionary pathway that will lead to a vertebrate eye where the nerves run behind the retina. As a result every last vertebrate has a blind spot where all of the nerves dive through the retina. This makes our eyes less efficient because the nerves block out light.
We're therefore stuck with the first design of the vertebrate eye and can't change it. There is no pathway to the more sensible design of the mollusk eye.
This is but one of many examples. For another one, relative developmental timing of growth is fixed across vertebrates. For example the "hand" grows before the "arm". The result is that pterodactyls, whose wings are entirely hands, could fly from birth. But birds (whose wings are arms) and bats (a mix of the two) can't fly from birth. No matter how desirable an evolutionary trait that may be.
No. At least not in the way described. Optimal as considered by whom? Us? We look at something and say this would function better than that, but is something that is optimal perfect? Basic microeconomics shows that perfection is sub-optimal due to the law of diminishing marginal return. Perfection is wasteful and unsustainable. What do we know to be optimal? From what I know, “adequate enough to reproduce” may not be perfect, but it might be brushing up against optimal in its “good enough and no more” / lagom nature. Mollusks have mollusk eyes. Can we know that humans would be better off with mollusk eyes without rewriting our evolutionary lineage for us to be more like mollusks in other imperfect ways?
Eyeballs are nowhere near optimal either.
I'm not necessarily against that as a valid way to measure job performance if it's done intelligently (for one, realizing that there's a lot of blind luck and variance, and it takes time to smooth out) - I mean, if you're in sales and you're not booking any sales, you don't get to hide from that. But it's also really easy to get ranking and evaluation schemes wrong, so I understand why people would be nervous about it and prefer soft-skill-based evaluations instead.
The upshot of this is that if Netflix or Facebook _do_ see a slow drift of users away from the platform, they are institutionally incapable of understanding why. Engagement is not the same thing as satisfaction, which is far harder to measure.
(I'm also now wondering how static the results of A/B testing really are - if an idea fails AB even once does the company abandon it forever?)
So I'm pretty sure that they are institutionally incapable of understanding why for a very good reason.
I feel like the general trend of UI simplification and a/b testing everything has companies chasing after anything that will push their general metrics up in the short term, at the expense of slowly disenfranchising a large portion of their users.
I interviewed at Netflix recently for a position on their UI team, and I questioned the interviewer about a lot of annoying aspects of the Netflix interface. He was aware of every complaint, and his explanation for every one of them was he wishes he could change it but the a/b testing shows they get higher engagement with it the way it is. So, for example, they show you movies you have already said you don't like, because enough people give a thumbs down and then later watch the movie. In the short term very few people are going to cancel Netflix because of this, but enough of these decisions in succession is erosive to customer goodwill.
Personally, these kind of decisions wouldn't bother me that much if apps and websites made it easy to customize your experience. Facebook makes it difficult by opaquely shaping your newsfeed and giving you what seems like hundreds of options across a half dozen settings pages. Netflix makes it difficult by just giving you practically no options, but there's enough demand that's there a tiny industry in just making websites and plugins to make Netflix usable.