News for hackers. Stuff that matters.
Re:Netflix never used its $1M algorithm (2012) by taeric on Wednesday January 10, 2024 @04:21PM (#38943342)
I'm curious how you quantify that?
Re:Netflix never used its $1M algorithm (2012) by gen220 on Wednesday January 10, 2024 @04:32PM (#38943503)
I'm not sure you need to.

Not picking on you at all, just going on a tangent because I've been thinking recently about the good ideas that end up on the cutting floor because their impact, while positive, is intractable or completely impractical to quantify.

Some ideas (like this Netflix one!) are "obvious" winners, because they have a diffused, positive impact in many important dimensions – each of which is almost impossible to measure, but the integral of which is almost certainly greater than the idea's cost, probably by one or two OOM. If there's high conviction in an idea being a 10x idea, and attempting it costs quite little, it's better to do it now and maybe consider measuring it later. Who cares if it's 9x or 12x ROI, the point is it has a big margin of safety and large expected returns on capital.

But in a "we can't greenlight this project unless we can directly attribute it to positive motion in our KPI" org, these flavors of idea are dead on arrival. The double-kicker is that the cost of trying to measure these ideas is often greater than the cost of the idea.

For some ideas, especially when it's a rounding error on the company's annual budget, it should be OK if we don't invest much effort into quantifying the results. It's one of those rare set of ideas where it's anathema to the professional/investor culture of SV of the last decade, while simultaneously being how plucky Seed/Series A companies can punch above their weight.

Re:Netflix never used its $1M algorithm (2012) by chmod775 on Wednesday January 10, 2024 @05:19PM (#38944195)
Yup. $1m is nothing. We live in a world where movies have $100m marketing budgets, game companies invest hundreds of millions into esports events and other material, and tiny companies are bought out "just in case" for similar amounts.

$1m to generate some publicity and attract talent? Sure. They're probably paying headhunters similar amounts in comissions each year.

Re:Netflix never used its $1M algorithm (2012) by taeric on Wednesday January 10, 2024 @07:00PM (#38945418)
Movies have very real returns that are trivial to see from box office results and merchandizing. Not to mention they have famously perverse incentives to spend as much of the profit from a movie as they can, due to how a lot of the royalties are structured.

More to the point, though, this is a bit of a tangent/deflection from my question. How do you assert the return on this $1m? This line simply says that no return was needed for it to be spent. It shows nothing to say how you calculate what is made from it.

Re:Netflix never used its $1M algorithm (2012) by taeric on Wednesday January 10, 2024 @04:38PM (#38943589)
I would push back on this specific case, though. What makes this an obvious winner? Netflix was already clearly on the upward swing. At the time of this contest, I'm not even sure I remember who their competition was.

Now, I think it is fully fair to say that this was very cheap for them to do. In which case, why not do it? But I think you would be hard pressed to give me a counter factual world that is believable where not doing this had a meaningful impact on Netflix's future.

Re:Netflix never used its $1M algorithm (2012) by gen220 on Wednesday January 10, 2024 @06:06PM (#38944811)
> I think you would be hard pressed to give me a counter factual world that is believable where not doing this had a meaningful impact on Netflix's future.

Some scale for perspective: if this had a 10x ROI, they'd get $10mm for $1mm. At EOY 2009 (same year this competition was paid out), Netflix had a twelve-month-trailing Revenue of $1.671B. In 2006 they took in ~$1Bn. So it's 0.06% of annual revenue at payout time, 0.1% of annual revenue at announcement/green-light time.

The idea had a large margin of safety: in the worst case, nobody pays attention and you've generated some IP which would have cost ~$1mm to generate anyways, and which would theoretically increase retention (a top 3 KPI) by X%. And it had big upside potential, in the form of additional marginal hires, press/exposure of the core product in the news cycle leading to marginal subscribers.

Re hiring: I think it's totally plausible that it contributed to ~5% of 500 peoples' employment decisions since 2009.

Re subscribers: it's plausible to think it contributed ~1% to 10k peoples' decisions to subscribe, most of whom have retained to-date. It's classic "Brand Advertising" – and although it's difficult to quantify, it works.

Those numbers could be way higher, but it doesn't really matter if they were or weren't, the margin of safety, uncapped upside of the brand advertising angle, and low relative cost made it a bona fide good idea.

Edit: FWIW, that we're still discussing this idea in 2024 is more evidence that it was a good idea. This whole thread is full of Netflix's target demographic. We're all talking about how well-compensated and smart Netflix engineers are... You would never be able to purchase that kind of long-burn, compounding, high-expected-value attention for $1mm from any advertising network, in 2006-2009 or now.

Re:Netflix never used its $1M algorithm (2012) by taeric on Wednesday January 10, 2024 @06:56PM (#38945384)
Right, I think this is a good analysis. But, I challenge that they actually got these returns. Specifically from this effort. It is popular among the type of crowd that posts here in HN. I would be surprised if any of my non-tech family even knows it happened.

I think I would push on the ~5% of 500 people's employment decisions. With the sizes of the teams that competed, I'd not be too surprised that the teams involved were largely the extent of who was directly impacted by this contest. Everyone else was as happy as those of us that read any of the blogs/articles that they would publish about their infrastructure. Which is not nothing, but probably hard to spot the difference. (Ironically, I would argue that the raised morale of the people that publish the articles are probably more measurable? Which, I confess I don't think is a consistent position from me.)

And I stress that I agree it was a safe thing to do. I am even glad that they did it. I just don't know that you could really convince me they would not have had the revenue that they had without doing this contest.

Re:Netflix never used its $1M algorithm (2012) by sosborn on Wednesday January 10, 2024 @05:06PM (#38943999)
Their competition was the old outdated business model. As easy as it was to say that streaming was the future, they still had to convince millions of people to learn about and sign up for it.
Re:Netflix never used its $1M algorithm (2012) by taeric on Wednesday January 10, 2024 @07:02PM (#38945431)
This implies that a large number of subscribers are here because of this. I would hazard most subscribers don't know this even happened. Much less did it have any impact on a choice to sign up. That was driven much more directly by high speed internet access.
Re:Netflix never used its $1M algorithm (2012) by sosborn on Wednesday January 10, 2024 @08:11PM (#38946079)
I would argue that any PR helps drive subscriptions.
Re:Netflix never used its $1M algorithm (2012) by taeric on Wednesday January 10, 2024 @10:25PM (#38947136)
I wouldn't fully disagree, but I would point out that they had a ton of press at the time for virtue of who they were. Indeed, a large part of why we still talk of them doing this contest, is because of who they were and are.

More directly, though, if you were to plot subscriptions to Netflix over time across an unlabeled chart, would you be able to see this contest?

Re:Netflix never used its $1M algorithm (2012) by sosborn on Thursday January 11, 2024 @01:19AM (#38948476)
I guarantee you they got much more than $1M worth of media placement for this stunt.
Re:Netflix never used its $1M algorithm (2012) by taeric on Thursday January 11, 2024 @10:26AM (#38953477)
If we are only referring to the cost of any media placement, I can't really disagree. My assertion is fully on ROI of this.

Take my challenge. Plot their subscribers and revenue over time, hiding the year/month. See if you can spot any sort of inflection where this happened.

It /could/ be that they would have seen declining numbers absent this. Such that this prevented a bad inflection on the chart. I don't know that I find that compelling.

Re:Netflix never used its $1M algorithm (2012) by sosborn on Thursday January 11, 2024 @11:07AM (#38954095)
Media placement is the long game and matching ROI to a single event is a fool's errand.
Re:Netflix never used its $1M algorithm (2012) by taeric on Thursday January 11, 2024 @12:21PM (#38955332)
To an extent I can't but agree. I can't help but speculate you are in the media placement game, though? :D

Like, I get that you can't measure everything. There are plenty of reasons to believe that.

I just can't bring myself to accept the odd logic that "popular tech company got press coverage which was instrumental in their growth." Seems there is a circular logic at play there that makes this such that, even if it is true, it is not actionable for anyone else.

Put another way, popular things get press coverage by virtue of them being popular things. At large, as long as they don't implode in very terrible coverage, they don't have to do much to get more coverage. Just doing anything will do that for them. To that end, odds are high we would still have discussed this even if they only offered something silly like a lifetime subscription for the winning team.

Re:Netflix never used its $1M algorithm (2012) by travisjungroth on Wednesday January 10, 2024 @04:44PM (#38943672)
Hiring. They got a lot of attention for this.
Re:Netflix never used its $1M algorithm (2012) by taeric on Wednesday January 10, 2024 @04:58PM (#38943882)
As I said in another reply, I don't remember them having trouble hiring in the first place?

I can certainly mostly agree with this. I just also think them producing their own shows was likely more impactful, and had basically nothing to do with this.

Don't get me wrong, plenty of moves in life are "win more." Such that I could see this one fitting that description. But a lot of "win more" choices are surprisingly low on the ROI side, when you control for everything else in that situation.

Re:Netflix never used its $1M algorithm (2012) by travisjungroth on Wednesday January 10, 2024 @05:55PM (#38944663)
Coca-Cola doesn't have trouble selling soda, but they still have commercials.

Of course making their own shows was more impactful in a bunch of ways. But, the investment is also so much more. A season of a show is in the tens to hundreds of millions.

Re:Netflix never used its $1M algorithm (2012) by taeric on Wednesday January 10, 2024 @06:50PM (#38945324)
I think that is a bit of an apples to oranges comparison. Coke does brand management. They are not necessarily educating folks to their existence.

But, the real difference is that Coke is, in fact, selling their own brand and products. For Netflix, they are largely selling access to other people's products. In a very real way, Netflix found an arbitrage opportunity for viewing movies. With how poorly the content owners calculated things, they got good deals at the beginning so that they could do well on their investment in the CDN infrastructure so that they could deliver a good experience to users with minimal unit costs.

This is, in a large sense, why Netflix is having to raise prices now. The content owners are starting to demand more money for that access, driving up their unit costs. Combined with a draught of original content, expect that they will lose access to their own productions in time. You can already see this with the likes of Daredevil.

Re:Netflix never used its $1M algorithm (2012) by travisjungroth on Wednesday January 10, 2024 @07:11PM (#38945511)
I'm talking about hiring. I believe this project had positive ROI for Netflix (where I used to work) because of how it helped bring attention to Netflix's ML problems. Maybe you disagree with this, totally fine.

My comment about Coca-Cola was to draw the analogy that just because they don't have trouble hiring (which you mentioned) doesn't mean it can't be helped.

Re:Netflix never used its $1M algorithm (2012) by taeric on Wednesday January 10, 2024 @07:18PM (#38945562)
Ah, that makes sense. I was thinking you were comparing to general advertising and the impact on drinker/user base.

If you saw a measured rise in hiring, I will trust you on that. My assertion here is largely on the trajectory that Netflix seemed to already have and how there is no clear "inflection point" in any of their numbers post this contest that I have been presented with. And probably more cynicism than makes sense for how little impact ML seems to have on any choices that Netflix seems to show.

Re:Netflix never used its $1M algorithm (2012) by kamkazemoose on Wednesday January 10, 2024 @04:30PM (#38943467)
You can think, how much would their marketing team have to spend to get the same results that the algorithm contract gave. I'm not a marketing expert, but I'm sure they have metrics like consumer sentiment, name recognition, number of users visiting the site, google search trends, etc. There could also be benefits in recruitment, and that can be estimated based on how much you'd have to pay an external recruiter to bring in candidates the applied, or other things like that.

It was in the news a lot, and was discussed on a lot of tech sites. Plus it gets people talking about their recomendation algorithm, and makes people thing Netflix subscription is more valuable becasue it recommends good shows. It wouldn't be cheap to get the amount of media that they got through more traditional marketing.

Re:Netflix never used its $1M algorithm (2012) by taeric on Wednesday January 10, 2024 @04:34PM (#38943541)
Right, but that is a lot of what I'm asking. What makes you think those are all better due to this contest? If I recall, a lot of why it was making the news was because Netflix was already popular in the industry. They certainly weren't that new of a name.

The main one I don't think I would have doubts about is the recruitment. But, I don't recall them being a place that needed recruitment help, even at that time.

To be clear, I found the thing fun to consider. I certainly am not upset that they did it. I do harbor a gut feeling that its ROI is greatly overstated.

Re:Netflix never used its $1M algorithm (2012) by kamikaz1k on Wednesday January 10, 2024 @04:26PM (#38943406)
at least for the internal goal of hiring top ML talent, you can probably just calculate how long it took to hiring the good ones, and then how much money that team made.
Re:Netflix never used its $1M algorithm (2012) by taeric on Wednesday January 10, 2024 @04:28PM (#38943440)
Are they objectively better at ML for movie recommendations than they were beforehand? I confess I've never been too impacted by their recommendations. Outside of new releases, but that doesn't really use any ML.