Why Netflix Never Implemented The Algorithm That Won The Netflix $1 Million Challenge
from the times-change dept
You probably recall all the excitement that went around when a group finally won the big Netflix $1 million prize in 2009, improving Netflix's recommendation algorithm by 10%. But what you might not know, is that Netflix never implemented that solution itself. Netflix recently put up a blog post discussing some of the details of its recommendation system, which (as an aside) explains why the winning entry never was used. First, they note that they did make use of an earlier bit of code that came out of the contest:A year into the competition, the Korbell team won the first Progress Prize with an 8.43% improvement. They reported more than 2000 hours of work in order to come up with the final combination of 107 algorithms that gave them this prize. And, they gave us the source code. We looked at the two underlying algorithms with the best performance in the ensemble: Matrix Factorization (which the community generally called SVD, Singular Value Decomposition) and Restricted Boltzmann Machines (RBM). SVD by itself provided a 0.8914 RMSE (root mean squared error), while RBM alone provided a competitive but slightly worse 0.8990 RMSE. A linear blend of these two reduced the error to 0.88. To put these algorithms to use, we had to work to overcome some limitations, for instance that they were built to handle 100 million ratings, instead of the more than 5 billion that we have, and that they were not built to adapt as members added more ratings. But once we overcame those challenges, we put the two algorithms into production, where they are still used as part of our recommendation engine.Neat. But the winning prize? Eh... just not worth it:
We evaluated some of the new methods offline but the additional accuracy gains that we measured did not seem to justify the engineering effort needed to bring them into a production environment.It wasn't just that the improvement was marginal, but that Netflix's business had shifted and the way customers used its product, and the kinds of recommendations the company had done, had shifted too. Suddenly, the prize winning solution just wasn't that useful -- in part because many people were streaming videos rather than renting DVDs -- and it turns out that the recommendation for streaming videos is different than for rental viewing a few days later.
One of the reasons our focus in the recommendation algorithms has changed is because Netflix as a whole has changed dramatically in the last few years. Netflix launched an instant streaming service in 2007, one year after the Netflix Prize began. Streaming has not only changed the way our members interact with the service, but also the type of data available to use in our algorithms. For DVDs our goal is to help people fill their queue with titles to receive in the mail over the coming days and weeks; selection is distant in time from viewing, people select carefully because exchanging a DVD for another takes more than a day, and we get no feedback during viewing. For streaming members are looking for something great to watch right now; they can sample a few videos before settling on one, they can consume several in one session, and we can observe viewing statistics such as whether a video was watched fully or only partially.The viewing data obviously makes a huge difference, but I also find it interesting that there's a clear distinction in the kinds of recommendations people that work if people are going to "watch now" vs. "watch in the future." I think this is an issue that Netflix probably has faced on the DVD side for years: when people rent a movie that won't arrive for a few days, they're making a bet on what they want at some future point. And, people tend to have a more... optimistic viewpoint of their future selves. That is, they may be willing to rent, say, an "artsy" movie that won't show up for a few days, feeling that they'll be in the mood to watch it a few days (weeks?) in the future, knowing they're not in the mood immediately. But when the choice is immediate, they deal with their present selves, and that choice can be quite different. It would be great if Netflix revealed a bit more about those differences, but it is already interesting to see that the shift from delayed gratification to instant gratification clearly makes a difference in the kinds of recommendations that work for people.
Thank you for reading this Techdirt post. With so many things competing for everyone’s attention these days, we really appreciate you giving us your time. We work hard every day to put quality content out there for our community.
Techdirt is one of the few remaining truly independent media outlets. We do not have a giant corporation behind us, and we rely heavily on our community to support us, in an age when advertisers are increasingly uninterested in sponsoring small, independent sites — especially a site like ours that is unwilling to pull punches in its reporting and analysis.
While other websites have resorted to paywalls, registration requirements, and increasingly annoying/intrusive advertising, we have always kept Techdirt open and available to anyone. But in order to continue doing so, we need your support. We offer a variety of ways for our readers to support us, from direct donations to special subscriptions and cool merchandise — and every little bit helps. Thank you.
–The Techdirt Team
Filed Under: contest, data, recommendation algorithm, streaming
Companies: netflix
Reader Comments
Subscribe: RSS
View by: Time | Thread
Sure there's a difference
The streaming algorithm, on the other hand, only has to achieve, say, a 5:1 recommendation:success ratio because if you get recommended five movies, stream two or three minutes of each, hate four and only watch one, then you've still watched a good movie, and you're probably satisfied.
It's a transformation from customer satisfaction correlating directly with the recommendation algorithm under a rental business model to a much looser correlation under the streaming model.
[ link to this | view in chronology ]
Re: Sure there's a difference
[ link to this | view in chronology ]
Re: Re: Sure there's a difference
[ link to this | view in chronology ]
This is very much the case for me. I use Lovefilm over in the UK (similar to Netflix) and there is a film I keep adding to my queue, they keep sending me and I keep sending back. I really want to watch it but I'm just never in the right mood but I hate holding onto the DVDs long term so send them back in short order.
Also looking at my queued films vs what I watch on instant there is a noticible difference. I watch a lot of trash on the instant player, films that I would never buy but that I'm happy to try out of curiosity at that moment. The queue though is full of films which are rated well and which I generally look forward to receiving in the post.
Obviously there is also the consideration that some films which I would watch on instant if available are not there so end up queued.
[ link to this | view in chronology ]
Re:
[ link to this | view in chronology ]
People work for free under the guise of a contest, and the best work is chosen and the author is given some bullshit prize for placation.
Designers are the ones who fall for it, given lack of experience, but programmers and computer scientists?
[ link to this | view in chronology ]
Re:
[ link to this | view in chronology ]
Re:
[ link to this | view in chronology ]
Re: Re:
I remember reading about this prize when it was still in competition and at least a few of the challengers didn't seem to think they had a chance to win, but kept working on it making incremental progress.
[ link to this | view in chronology ]
Re:
What's interesting about this is that their suggestion system for streaming videos still affects the company's value prop. And the whole point of the machine learning algorithm is that based on a sample set of millions of input data, they can extrapolate a rating system that holds true for billions of future cases (at least better than their current system).
What I'm saying is they have even more data now, and they could probably pay some engineers to build a new machine learning algorithm for streaming videos, with a very good chance of making a much more accurate system. I'd rather see them spend money on that then trying to launch their own cable channel.
[ link to this | view in chronology ]
Re: Re:
I believe when you are coming up with a machine learning algorithm, it is considered bad practice to personally look at the data set you are given. You really only need to know the format and size of the data.
If it was a more collaborative effort, they could have worked with the research teams and basically run the algorithms themselves, giving results back to the teams. This way, the data would never leave Netflix, and there would be no privacy issues.
[ link to this | view in chronology ]
$1,000,000 is bullshit?
[ link to this | view in chronology ]
Made me immediately think of Natalie Tran's observations on current self vs past self. If you haven't seen the video it's here:
http://www.youtube.com/watch?v=zRyf23boUO4
One of the most practical KDDCUP competitions - a pity the competition didn't match more closely the engineering requirements.
[ link to this | view in chronology ]
I think they have it nailed.
I usually watch them one skit at a time. I hope they don't take partially watched to be evidence that I don't like it, it's just that that sort of comedy is better in small doses.
[ link to this | view in chronology ]
Canadian selection
I wish when i searched for a streaming movie, that it could play the movie trailer.
netflix is like searching through the discount dvd bin at walmart.
[ link to this | view in chronology ]
Re: Canadian selection
I find Netflix's recommendations to be excellent, but then I've rated 3,680 movies.
[ link to this | view in chronology ]
Re: Canadian selection
[ link to this | view in chronology ]
Actually, that makes sense
My initial response was to say that they spent $1 million getting a better algorithm and they didn't use all of it, what were they thinking?
But then, if it didn't make sense to use it when they finally got the algorithm then don't use it. It would be stupid in this situation to use it.
[ link to this | view in chronology ]
Garbage in, garbage out
Netflix could have fixed this years ago about allowing multiple users under one account, but they never cared. That's why the ratings on Netflix have turned to crap, and (that lack of caring) is why Netflix is itself turning to crap.
[ link to this | view in chronology ]
Re: Garbage in, garbage out
[ link to this | view in chronology ]
management essay
management essay
[ link to this | view in chronology ]
steaming netflix movies cannot be recomended accurately
The movies they offer for streaming could more accurately be offered steaming, as in a steaming pile of dog poop. Until they consistently offer movies I'd be interested in, they can keep their steaming content; I don't want it.
[ link to this | view in chronology ]
competition
[ link to this | view in chronology ]
[ link to this | view in chronology ]
This was never implemented because it did not do as it was planned to do as an algorithm .
[ link to this | view in chronology ]