in Reed's "No Rules Rules" book, they discuss how this contest was really a way to recruit top-quality engineering talent, which is a key assumption in how they ran their culture (highly paid small teams with lots of freedom, trusted to know what's best from the ground-up rather than top-down).
They didn't really need an algorithm in the first place, they just brainstormed what would sound the coolest to developers.
A little secret is that the content rights holders negotiated draconian streaming contracts that are outright prohibitive to Netflix. Netflix is trying to do everything possible to hide the fact that their streaming library is small and shrinking. In the old hey-day of Netflix mail-in red envelopes with DVDs you were hard pressed to find a movie they did not have. These days your search returns mostly crap.
> A little secret is that the content rights holders negotiated draconian streaming contracts that are outright prohibitive to Netflix. Netflix is trying to do everything possible to hide the fact that their streaming library is small and shrinking.
I think that's fairly well-known at this point, though I agree it should be more widely known.
I think the real secret is that Netflix discovered that their users aren't trying to watch the best movies, they're just trying to kill time. Apparently you can substitute good films for generic junk and people will keep their subscriptions.
Really, the business model for Netflix has some weird perverse incentives. They make money from monthly subscriptions. They lose money when actually streaming content to users. So the ideal user from their standpoint is one who keeps paying but never watches anything.
Thus, the company is incentivized to put out just enough good stuff that you can't miss that you don't cancel, but otherwise produce garbage you don't want.
This is why I'm still surprised they separated out (and eventually killed) their disc-by-mail service. I understand the visceral feeling that discs are an antiquated relic—but you just can't beat that catalog.
Combining discs and streaming in one subscription provides customers with the best of both worlds. Need something to watch right now? No problem, we have an excellent streaming catalog and you're going to love Stranger Things! Want a specific movie or show, maybe Friends or The Lion King? We've got that too, it'll be in your mailbox in a couple days.
And if some customers choose to forgo either the digital or physical side of the package, it doesn't cost Netflix anything.
Funny enough it has been so long that we have come full circle on this. Studios that were pulling their content from Netflix to launch competing streaming services are now willing to license it again because their services are failing, leaving them bleeding for cash. You can even find the newest Disney and Warner Bros movies and shows on Netflix now.
The general consensus in the media industry is that Netflix basically won the streaming wars this year. Every other service is contracting, has a fraction of the viewership, is running at a loss. Netflix is running in the black, has by far the highest viewership (like an order of magnitude over many of them), dominates over seas. The other studios have started to license content again to get every last bit of cash flow they can.
My favorite part about this is that a lot of the rights holders are losing their shirts on their alternative streaming services, because it turns out making content is different than running a profitable streaming service. Peacock and Disney+ are losing huge money, Amazon is introducing commercials or an upcharge to their service because they've been spending on their own content with abandon (I can't imagine why WoT costs per episode what I read it costs), and I don't know how appletv+ is doing in 2023 but in 2022 and 2021 they were also losing huge sums of money. Is netflix the only major streaming service that makes money at this point?
Everyone decided "I can cut out the middleman and keep the money in my pocket" and they all lost hugely on that strategy.
The real "Netflix Optimization Problem" is them figuring out how to minimize the cost of licensing or creating content while staying just good enough that you renew your subscription.
I'm still surprised Netflix hasn't bought a bunch of media companies with big back catalogs. I feel like it'd help with people complaining about lack of content and give them better negotiating power.
> Netflix is trying to do everything possible to hide the fact that their streaming library is small and shrinking.
And it appears to be working. I was telling this girl the other day about why I canceled Netflix years ago, because they frequently didn't have whatever I searched for. She insisted that "no, Netflix has everything!" So we proceeded to search for various things with her account, including the final straw that made me cancel. Netflix proceeded to not have a single thing we searched for.
There are a couple movies/TV shows that were unobtainable via Netflix DVDs, and which seem unstreamable. Some very old DVDs float around at high prices.
Examples are: Exit to Eden, The Others, and Baccano!
How does this happen? Why do rights holders want movies killed?
All the top talent they've acquired and money spent still can't save them from their complete garbage lineup of content they push out, all while limiting account access and raising prices.
Neflix revenue went from 4.3b in 2013 to 31b in 2022. Astounding success by any measure. That means they are doing something right, including their talent strategy. "garbage" is subjective. Neflix employees are likely paid bonuses on views / subscriptions generated, which from their revenues looks like they know how to hit.
Netflix obviously since enjoyed a period of success. You may need to reconcile that with your overly negative portrayal of the situation which honestly seems more about your personal feelings about Netflix rather than more objective measures of success.
You call it garbage content, but every other person is somehow watching them. I’m probably not their target audience, but looking at my friend circles, those shows are definitely being consumed.
Unlike the old days of broadcast TV when the networks had to produce shows that appealed to the broadest possible audience, Netflix can, and does, produce shows that appeal to a narrow demographic. This results in a sliver of shows that are highly appealing amidst a lot of stuff perceived as "garbage." But what a teenage boy thinks is garbage differs widely from what a retired grandma things is garbage. To each their own.
I kind of agree with you. I feel like there's a narrow set of content there worth watching and when you get through it all it's just like: okay, now what? There's nothing else worth watching. I haven't been able to use my netflix account for months now. I just go to youtube now and subscribe to channels I like. Seems much more interesting than netflix.
Currently following some interesting permaculture projects. I find the type of work these people do so useful and clever really.
It's funny, all the top talent, all the blog posts, yet sorry to say as everyone has long learned, streaming platforms are utter commodity and were not an unique advantage at all. All made up. They were always just a nice wrapping paper around their content & licensing.
Wait, so they knew _in advance_ that they weren't going to use the result? Before they even started the competition? Or is this ret-conning once they decided not to use it. I feel like I can't really trust Reed's word on it, since he has a very strong incentive to paint himself as a mastermind.
In these kind of competitions (e.g. Kaggle), the algos never really get used because they are too complex/optimized to implement into production (i.e. 3 layers of 10 ensembled XGB + 1000 highly engineered features + etc, etc).
Mainly they are used for recruiting, new ideas (simpler versions get implemented), and proof of concept (if we did this really really well, what would it look like, should we spend 2 engineering years on it?)
Everything that Netflix says or does externally wrt to tech appears to be for this purpose, including open-sourcing some things. It is all catnip for the HN crowd to come work for them.
Netflix pays in the same ballpark as other FAANG companies but the product is so much clearer (stream people movies and shows, maybe recommend them something to watch) and the complete lack of grey ethics like in AdTech or a Monopoly like Microsoft is the real catnip.
So the entire premise by which they were “recruiting” was dishonest? No one sees any problems with lying to people? Dishonesty has apparently become the norm, not the exception. Honestly and trust is a bedrock principle of society. It’s no wonder our society is in the state that it’s in today.
You really cannot trust anything anyone says, especially large companies. They basically always lie and manipulate and are at a constant advantage to those of us who find dishonesty morally wrong.
Eh, I don't think anyone participated because they just really wanted Netflix to have a better recommendation system. Teams put serious effort into it for the million dollar prize, which wasn't a lie. And hackers and tinkerers enjoyed the novelty of the challenge and the cool dataset to work with, also not a lie.
I don't think they ever said they would use it necessarily either? I feel like it was obvious that it wasn't going to capture all of their actual requirements - your algorithm could be slow or expensive or difficult to adapt to new requirements or difficult to adapt to new data etc etc.
Ironically, Netflix's recommendation algorithm is now notoriously bad. In my experience, it seems to heavily push whatever "original" they just dumped $100 million into as well as what's most popular on the platform at the time. But it makes sense, since people increasingly rely on social media for deciding what to watch. A dollar Netflix spends improving their recommendation algorithm simply won't stack up to the dollar TikTok/Instagram/etc. spends improving theirs, because social media apps have so much more data to work with. It's probably more economical from their POV to let broad social media trends dictate what people watch.
I suspect their algorithm has other success metrics than "did user enjoy movie". For example if the movie is a Netfix Original or exclusive movie then it is more valuable to Netflix because they may discuss or recommend it, which leads to more signups. Similarly they may prefer newer shows that are more likely to be discussed and raise hype than older movies that even if the user loves them may be less likely to advertise to others.
This 100%. I promise you that I’m not secretly very interested in Dragon Rider episodes without my kids present. They have their own accounts, but it is hard to share family movies or when they just use the wrong account.
I also think half of my viewing is with my spouse. We are pretty aligned in some shows but we are different enough where “My List” really means when I’m alone, and we are missing an “Our List” for when we are together.
As a sibling pointed out. This already exists. Profiles don't just let you do "John" they let you do "John and Sally". Maybe they should be more explicit that people can be creative ith their profiles. Would be good marketing for people who are stuck like this.
I'm pretty happy with how the accounts work: my children watch Octonauts in a Kids account (sometimes I watch with them), my wife and I have separate accounts, so she doesn't get recommendations for war movies and psychological thrillers, and I don't get recommendations for Queer Eye for the Straight Guy.
in my experience from the outside it's difficult to separate their rec algorithm quality vs. the quality of their inventory. you're making the assumption they have a ton of great stuff that they're just not showing you, but they may not.
I wonder if the real goal is the secondary effect. If you watch a show on Netflix and enjoy it, that’s good for customer satisfaction. If you watch their exclusive original content, not only are you satisfied but now you are telling all your friends about something only Netflix can provide.
As someone who works in search relevance, having just a great algorithm isn't worth much.
You need all the team and know-how that has the maturity to maintain such an algorithm. Not just the ML skills. But all the bazillion ops, data quality, and many other things that go around it.
I've worked with a lot of teams that have one smart person building stuff off to the side, in R or a Notebook, and then nobody knows how to productionize it. They try to throw the algorithm over the fence. Even if the team somehow succeeds in getting it into an A/B test, it eventually falls by the wayside, unless they can build the team and workflow around that person / algorithm / methodology
> I've worked with a lot of teams that have one smart person building stuff off to the side, in R or a Notebook, and then nobody knows how to productionize it. They try to throw the algorithm over the fence. Even if the team somehow succeeds in getting it into an A/B test, it eventually falls by the wayside, unless they can build a team and workflow around that person / algorithm / methodology.
In my view, this marks a cultural failure - and certainly not on the part of the 'one smart person.'
Using a recommendation algorithm doesn't make any sense for Netflix' new business model where they produce their own content. It should be easy. They greenlight or buy the rights certain content for different demographics to maximize the coverage of their user base that has their needs satisfied enough to stay subscribed. If you're a 18-35 male, they've surely got people there working daily to make sure there's a content pipeline coming just for you. They shouldn't need AI to tell me that, they just need to identify when I'm in the demographic of one of their shows coming, and tell me about it. Maybe like a list. Still, they seem like they're doing a bad job at it, as I never really know what new shows Netflix is making for me. Sometimes I find out about them years later.
For example, compare this to Disney+. I vaguely know about every Marvel and Star Wars thing 2 years in advance, and usually vaguely know what order they're coming out, and I don't even know how I know this, somehow I just know. Okay, that's easy because it's basically 2 IPs. But ultimately Netflix is doing something similar behind the scenes, why haven't they succeeded at making me aware of what content I'm supposed to be hyped for?
To be honest, I would be happy if their 20+ people UX team would create an experience where I can easily find what I was watching instead of shoving stuff down my throat that I have no interest in.
Then again, as long as people pay for that experience, it will continue to be as unbearable as it is.
Right? Like can we please get a user story on some agile board somewhere that says: "As a user, if the last 10 times I opened netflix I watched the same show, I should be shown the option to resume that show very prominently"
The number of times I've had to scroll or even search to find the thing that I very obviously want seems intentionally maddening.
This is the problem across the board, regardless of service almost. Everything is some crappy recommendation system, instead of enabling me to find what I want.
In addition to sucking in general, recommendation systems have this great property of radicalizing people politically.
This feels like a disingenuous argument. “Young to middle-aged man likes Marvel and Star Wars” is such an obvious pick that it just about invites parody. As someone with at this point zero interest in Star Wars / Marvel content I a) wouldn’t retain the information that you did and b) wouldn’t find it to be that impressive that D+ was shoving it into my face.
To not understand the usefulness of recommendation algorithms almost feels intentionally contrarian.
Not picking on you at all, just going on a tangent because I've been thinking recently about the good ideas that end up on the cutting floor because their impact, while positive, is intractable or completely impractical to quantify.
Some ideas (like this Netflix one!) are "obvious" winners, because they have a diffused, positive impact in many important dimensions – each of which is almost impossible to measure, but the integral of which is almost certainly greater than the idea's cost, probably by one or two OOM. If there's high conviction in an idea being a 10x idea, and attempting it costs quite little, it's better to do it now and maybe consider measuring it later. Who cares if it's 9x or 12x ROI, the point is it has a big margin of safety and large expected returns on capital.
But in a "we can't greenlight this project unless we can directly attribute it to positive motion in our KPI" org, these flavors of idea are dead on arrival. The double-kicker is that the cost of trying to measure these ideas is often greater than the cost of the idea.
For some ideas, especially when it's a rounding error on the company's annual budget, it should be OK if we don't invest much effort into quantifying the results. It's one of those rare set of ideas where it's anathema to the professional/investor culture of SV of the last decade, while simultaneously being how plucky Seed/Series A companies can punch above their weight.
You can think, how much would their marketing team have to spend to get the same results that the algorithm contract gave. I'm not a marketing expert, but I'm sure they have metrics like consumer sentiment, name recognition, number of users visiting the site, google search trends, etc. There could also be benefits in recruitment, and that can be estimated based on how much you'd have to pay an external recruiter to bring in candidates the applied, or other things like that.
It was in the news a lot, and was discussed on a lot of tech sites. Plus it gets people talking about their recomendation algorithm, and makes people thing Netflix subscription is more valuable becasue it recommends good shows. It wouldn't be cheap to get the amount of media that they got through more traditional marketing.
at least for the internal goal of hiring top ML talent, you can probably just calculate how long it took to hiring the good ones, and then how much money that team made.
The Netflix Prize competition (2006, completed in 2009) was a Kaggle competition before Kaggle competitions (2010), with the same business-side incentives.
Given the meteoric rise of ML/AI in the past few years, I'm surprised that Kaggle doesn't come up more often. It was all the rage 2013-2018...then most I heard about it was that it allowed free access to TPUs.
I feel like kaggle is a copycat wasteland for most projects now. You don't quite get top talent, rather you get a bunch of people xgboosting their way up your leaderboard.
"...rather you get a bunch of people xgboosting their way up your leaderboard" -->
I don't see that as a universally bad thing. I feel this is actually very representative of the majority of ML projects that most organizations encounter. Most organizations don't have petabytes of data and a huge compute budget to train a DNN. They typically have megabytes to gigabytes of somewhat crappy data and need something that can be developed and deployed relatively quickly for low cost.
There have been quite a few interesting Kaggle competitions in recent years, as well as other interesting ML/data science competitions on other platforms.
Platforms like Kaggle, DrivenData, Zindi, AIcrowd, CodaLab and others are running dozens-hundreds of competitions a year in total, including ones linked to top academic conferences. One interesting recent one is this one on LLM efficiency - trying to see to what extent people can fine-tune an LLM with just 1GPU and 24h: https://llm-efficiency-challenge.github.io/
Or the Makridakis series of challenges, running since the 80s, which are a great testbed for time-series models (the 6th one finished just last year): https://mofc.unic.ac.cy/the-m6-competition/
Interesting that the answer is basically "because of akrasia". I wish more of these recommendation algorithms were based more off of who we'd like to be rather than how we behave.
Anyone who has experience with ML wouldn’t be surprised by that. Oftentimes ML competitions are about combining dozens of models together to juice the extra 0.01%—something that isn’t viable in a production environment as the quote in the article confirms.
AFAIK, modern ML challenges try to combat that by moving from answer-only submissions to code submissions and putting constraints on compute.
Readit News
They didn't really need an algorithm in the first place, they just brainstormed what would sound the coolest to developers.
I think that's fairly well-known at this point, though I agree it should be more widely known.
I think the real secret is that Netflix discovered that their users aren't trying to watch the best movies, they're just trying to kill time. Apparently you can substitute good films for generic junk and people will keep their subscriptions.
Really, the business model for Netflix has some weird perverse incentives. They make money from monthly subscriptions. They lose money when actually streaming content to users. So the ideal user from their standpoint is one who keeps paying but never watches anything.
Thus, the company is incentivized to put out just enough good stuff that you can't miss that you don't cancel, but otherwise produce garbage you don't want.
Combining discs and streaming in one subscription provides customers with the best of both worlds. Need something to watch right now? No problem, we have an excellent streaming catalog and you're going to love Stranger Things! Want a specific movie or show, maybe Friends or The Lion King? We've got that too, it'll be in your mailbox in a couple days.
And if some customers choose to forgo either the digital or physical side of the package, it doesn't cost Netflix anything.
Everyone decided "I can cut out the middleman and keep the money in my pocket" and they all lost hugely on that strategy.
And it appears to be working. I was telling this girl the other day about why I canceled Netflix years ago, because they frequently didn't have whatever I searched for. She insisted that "no, Netflix has everything!" So we proceeded to search for various things with her account, including the final straw that made me cancel. Netflix proceeded to not have a single thing we searched for.
Deleted Comment
Examples are: Exit to Eden, The Others, and Baccano!
How does this happen? Why do rights holders want movies killed?
Deleted Comment
Currently following some interesting permaculture projects. I find the type of work these people do so useful and clever really.
Mainly they are used for recruiting, new ideas (simpler versions get implemented), and proof of concept (if we did this really really well, what would it look like, should we spend 2 engineering years on it?)
Deleted Comment
I was just an outsider, but I guess at least I learned about passive levitation.
You really cannot trust anything anyone says, especially large companies. They basically always lie and manipulate and are at a constant advantage to those of us who find dishonesty morally wrong.
I don't think they ever said they would use it necessarily either? I feel like it was obvious that it wasn't going to capture all of their actual requirements - your algorithm could be slow or expensive or difficult to adapt to new requirements or difficult to adapt to new data etc etc.
Husband.
Wife.
Husband and Wife.
Husband and 4th grader.
Wife and Pre-K.
Husband, Wife, 4th grader, and Pre-K.
4th grader.
Pre-K.
4th grader and Pre-K.
Each of those has a totally different viewing pattern and preferences.
I also think half of my viewing is with my spouse. We are pretty aligned in some shows but we are different enough where “My List” really means when I’m alone, and we are missing an “Our List” for when we are together.
You need all the team and know-how that has the maturity to maintain such an algorithm. Not just the ML skills. But all the bazillion ops, data quality, and many other things that go around it.
I've worked with a lot of teams that have one smart person building stuff off to the side, in R or a Notebook, and then nobody knows how to productionize it. They try to throw the algorithm over the fence. Even if the team somehow succeeds in getting it into an A/B test, it eventually falls by the wayside, unless they can build the team and workflow around that person / algorithm / methodology
In my view, this marks a cultural failure - and certainly not on the part of the 'one smart person.'
For example, compare this to Disney+. I vaguely know about every Marvel and Star Wars thing 2 years in advance, and usually vaguely know what order they're coming out, and I don't even know how I know this, somehow I just know. Okay, that's easy because it's basically 2 IPs. But ultimately Netflix is doing something similar behind the scenes, why haven't they succeeded at making me aware of what content I'm supposed to be hyped for?
Then again, as long as people pay for that experience, it will continue to be as unbearable as it is.
The number of times I've had to scroll or even search to find the thing that I very obviously want seems intentionally maddening.
In addition to sucking in general, recommendation systems have this great property of radicalizing people politically.
To not understand the usefulness of recommendation algorithms almost feels intentionally contrarian.
Not picking on you at all, just going on a tangent because I've been thinking recently about the good ideas that end up on the cutting floor because their impact, while positive, is intractable or completely impractical to quantify.
Some ideas (like this Netflix one!) are "obvious" winners, because they have a diffused, positive impact in many important dimensions – each of which is almost impossible to measure, but the integral of which is almost certainly greater than the idea's cost, probably by one or two OOM. If there's high conviction in an idea being a 10x idea, and attempting it costs quite little, it's better to do it now and maybe consider measuring it later. Who cares if it's 9x or 12x ROI, the point is it has a big margin of safety and large expected returns on capital.
But in a "we can't greenlight this project unless we can directly attribute it to positive motion in our KPI" org, these flavors of idea are dead on arrival. The double-kicker is that the cost of trying to measure these ideas is often greater than the cost of the idea.
For some ideas, especially when it's a rounding error on the company's annual budget, it should be OK if we don't invest much effort into quantifying the results. It's one of those rare set of ideas where it's anathema to the professional/investor culture of SV of the last decade, while simultaneously being how plucky Seed/Series A companies can punch above their weight.
It was in the news a lot, and was discussed on a lot of tech sites. Plus it gets people talking about their recomendation algorithm, and makes people thing Netflix subscription is more valuable becasue it recommends good shows. It wouldn't be cheap to get the amount of media that they got through more traditional marketing.
Given the meteoric rise of ML/AI in the past few years, I'm surprised that Kaggle doesn't come up more often. It was all the rage 2013-2018...then most I heard about it was that it allowed free access to TPUs.
I don't see that as a universally bad thing. I feel this is actually very representative of the majority of ML projects that most organizations encounter. Most organizations don't have petabytes of data and a huge compute budget to train a DNN. They typically have megabytes to gigabytes of somewhat crappy data and need something that can be developed and deployed relatively quickly for low cost.
Platforms like Kaggle, DrivenData, Zindi, AIcrowd, CodaLab and others are running dozens-hundreds of competitions a year in total, including ones linked to top academic conferences. One interesting recent one is this one on LLM efficiency - trying to see to what extent people can fine-tune an LLM with just 1GPU and 24h: https://llm-efficiency-challenge.github.io/
Or the Makridakis series of challenges, running since the 80s, which are a great testbed for time-series models (the 6th one finished just last year): https://mofc.unic.ac.cy/the-m6-competition/
AFAIK, modern ML challenges try to combat that by moving from answer-only submissions to code submissions and putting constraints on compute.