Winner Take All Outsourcing: Netflix Will Pay $1 Million To Whoever Improves Their Recommendation Engine

from the and-the-rest? dept

Mon, Oct 2nd 2006 1:53am — Mike Masnick

The concept of crowdsourcing has been popular lately, but not everything is easily sourced out to the crowd. Netflix is trying their hand at the game by throwing a little money into the pot. They're offering a million dollars to anyone who can improve the accuracy of their movie recommendation engine. Of course, there are some questions here. They claim they'll award the prize to whoever makes their system 10% or more accurate -- but it's not clear (at least from the NY Times article) how "accuracy" is determined. While some people do seem excited that Netflix has released a very large dataset for people to work on, part of this seems like an attempt to get a lot of free labor by making it a contest. It's true that contests like the X-Prize for space flight and the Grand Challenge for autonomous vehicles helped spur a lot of innovation -- but both were in general areas where the research efforts could easily lead to entire new industries. With a better recommendation system, it's basically just to help Netflix. And, to be honest, it just doesn't seem as exciting to help Netflix help you pick a better movie using collaborative filtering as it does to shoot a rocket into space. Still, this is an interesting trend, and you have to wonder if other companies will start experimenting with this "winner takes all" form of outsourcing -- or if people realize that all of the losers end up with nothing. Companies have plenty to gain if such programs work out -- but it seems like plenty of people will question whether it's worth building new tools for a company when they might not even get paid for it.

Thank you for reading this Techdirt post. With so many things competing for everyone’s attention these days, we really appreciate you giving us your time. We work hard every day to put quality content out there for our community.

Techdirt is one of the few remaining truly independent media outlets. We do not have a giant corporation behind us, and we rely heavily on our community to support us, in an age when advertisers are increasingly uninterested in sponsoring small, independent sites — especially a site like ours that is unwilling to pull punches in its reporting and analysis.

While other websites have resorted to paywalls, registration requirements, and increasingly annoying/intrusive advertising, we have always kept Techdirt open and available to anyone. But in order to continue doing so, we need your support. We offer a variety of ways for our readers to support us, from direct donations to special subscriptions and cool merchandise — and every little bit helps. Thank you.

–The Techdirt Team

18 Comments

If you liked this post, you may also be interested in...

Reader Comments

Subscribe: RSS

View by: Time | Thread

web proxy, 2 Oct 2006 @ 2:24am

they must divide such a hude sum of prize money ... say top 100 developers .... thing .... and plus it should be more clear that how this "accuracy thing " is determined ?
[ link to this | view in chronology ]
RantMax, 2 Oct 2006 @ 5:00am

No way!
"While some people do seem excited that Netflix has released a very large dataset for people to work on, part of this seems like an attempt to get a lot of free labor by making it a contest."

You think?

As for accuracy, isn't it obvious: they'll only claim success if they have 10% boost of sales.

Of course that little thing called "customer budget" which may stop this from happening, even if the recommendation becomes extremely accurate.

Netflix's business model is based on quasi random actions in attempt to find what works by means of natural selection.

This is strangely familiar to what they are doing here.
[ link to this | view in chronology ]
Sanguine Dream, 2 Oct 2006 @ 5:16am

Yeah right...
And what is to stop them from taking in all the entries, inform all the entrants that none of them added the +!0% accuracy boost then a few months later they release a "new recommendation system". And of course it will just happen to be modified (Netflix may just have the balls or ovaries to not even modify it) code from one or more of the entrants. And one more thing...Netflix will make sure to patent it before they release it in order to prevent law suits from the developer(s) behind this "new system".
[ link to this | view in chronology ]
Nobody Special, 2 Oct 2006 @ 5:21am

other thoughts
1) If you word the contest correctly, you can take all the small incremental changes and simply not award the prize because no one person (or group) made the requisit amount of change.

2) What really makes it different from X-Prize and Grand Challange is that both of those areas had people already working on it. In both of those contests, the winners would simply ofset their expenses. The real prize is being the winner. The money is more about generating excitement in the public and thus increasing the amount of bragging rights.

3) I suspect what Netflix really wants is something else. While it would be great to brag about how accurate the suggestions are.... the biggest benefit would be instead to use said engine to decide how many of each movie to buy. They waste money anytime the buy more copies of a movie then warrented by customer demand.

In any business dropping costs has obvious benefits. But in their business if they can cut cost of movies by say 5% it would be a huge gain for the bottom line.
[ link to this | view in chronology ]
Trey, 2 Oct 2006 @ 5:58am

Holy cripes!
Has anyone here ever just gotten excited? You guys spend all day looking for the catch, because you are afraid someone is going to make fun of you! This isn't highschool anymore. It's a million bucks, if your good enough you may get yourself a large step twords retirement. So my 2 cent, but positive, shiny happy 2 cents.
[ link to this | view in chronology ]
joh, 2 Oct 2006 @ 6:49am

I don't know what they have in mind, but it should be easy to measure how accurate a recommendation system is. Just take a particular user, remove their rating for some movie, see what rating the engine predicts they would give that movie, and compare with what they actually gave it. Now repeat with a lot of different users/movies until you're satisfied you've got the average accuracy.

And I don't see that there's anything sinister here. If you work on it and decide in the end your work is worth more than a shot at the prize, you don't have to give it to them.
[ link to this | view in chronology ]
Craig, 2 Oct 2006 @ 7:04am

"Collaborative filtering" has a fairly substantial research base, especially in the computer science area. Many algorithms have been developed. The main problem, at least as I've come to understand it, is data sparsity: when you have 40,000 movies and 1.5 million customers, each customer has rated only like .01% of all films, making it _very_ hard to derive patterns among individuals that match well enough to make solid recommendations 90% of the time.
[ link to this | view in chronology ]
Dam, 2 Oct 2006 @ 7:07am

Here's My Improvement
Get the studios to release DVDs the same day as the theatrical release and then pay Netflix to run trailers on the site the way they do on TV. With all the hype surrounding most new movie releases, everyone's recommendation can include the same day releases. The studios and Netflix will l make money, and the customers will be happy to see a hot new flick now instead of waiting - or going into a theater - yuck!

Netflix - just credit my account with $1MM and I'll take it out in rentals for the next 50 years.
[ link to this | view in chronology ]
tim finin, 2 Oct 2006 @ 7:26am

Possible privacy problems
This dataset will be a great asset to researchers working on user modelling, machine learning and recommendation systems. There may be some privacy problems that surface, however. Given that you know I am a Netflx customer and a few unsuual moves that I've received from Netflix, can you pick out my profile? If so, it might reveal a few movies that I don't want the everyone to know I rated highly, like Gidget Goes Hawaiian. Shades of Bork!
[ link to this | view in chronology ]
JG, 2 Oct 2006 @ 9:04am

Could be worth a shot
Why not? It doesn't hurt Netflix to try. As noted, there is quite a bit of research in this field already, so it seems unlikely that anyone who is not already working on this will dream up a better solution, but who knows? And if you as a programmer think you would like to take a shot, more power to you! Just be careful...

Of greater interest to me is the fact that Netflix is trying to improve at all. Wired recently ran an article about Netflix sponsoring the independent film movement, and they mentioned in the article how much "better" their recommendation search was compared to sites like Amazon. Constant improvement is great, but maybe its not as great as they claimed?
[ link to this | view in chronology ]
Lloyd Fassett, 2 Oct 2006 @ 10:55am

PR is cheap, curve jumping results require more
On the good side, this generates great PR and it does open up a lot of data for modelers to work with. Best of all, for Netflicks, is that they only pay for results and don't have to guess at paying for people's time and managing them. They get a lot of effort and only pay for results. It's an important pay for performance model for the future.

On the bad side, the dataset is for collaborative filtering and they are looking for a way to find gold in their data. I sense enough good smart people have tried with collaborative filtering and the result for the prize is damn near impossible because anyone with depth in the practice would have been able to approach Netflix and Amazon and anyone else with their great algorithm anyway. And, if you did have that algorithm that could run on any collaborative filtering dataset, why would sell it for only $1million, when you could probably find 3 customers willing to pay that price?

This content and dataset format worked for finding gold in Canada because the problem posed and dataset were solvable within the given real world constraints. The spaceship prize worked because the result was demonstrable and the solution could come from anywhere.

This problem is different because to get 10% increase in accurracy, effectively jumping a curve, will require a new technology. This solution is looking for a new tool and that's not going to come out of the dataset. That will require money to be spent and different data to be collected, or even other data to be accessed than what is offered, which is where the contest is falling short.

What they need is the new new thing after Collaborative Filtering that explains this kind of phenomena: http://www.popvssoda.com/countystats/total-county.html

You can't get something for nothing, except PR...
[ link to this | view in chronology ]
Ex NetFlix User, 2 Oct 2006 @ 11:35am

Why improve the accuracy of the recommendation engine when you are only going to throttle the result?
[ link to this | view in chronology ]
- Jimbo, 2 Oct 2006 @ 12:33pm
  
  Re:
  Bravo! I got so fed up with Netflix's throttling that I finally dumped them. Here's an idea, maybe throttle those who don't rate the movies they rent?
  [ link to this | view in chronology ]
GrammarNazi, 2 Oct 2006 @ 12:05pm

I hate to say it, but..
..shouldn't it be "to whomever"?
[ link to this | view in chronology ]
innerdaemon, 2 Oct 2006 @ 12:21pm

Lookie here: another $1 million bounty
Intel offers $1 million for anyone who can come up with a design similar to the Apple Mac Mini.
http://crave.cnet.co.uk/desktops/0,39029426,49284049,00.htm
[ link to this | view in chronology ]
Anonymous sucker, 3 Oct 2006 @ 2:01pm

RE: yeah right
Limp Bizkut did the same thing! Get the collective talent of hundreds of people working for you. Have them sign away their rights.

Read about it here: http://www.greenspun.com/bboard/q-and-a-fetch-msg.tcl?msg_id=007w74
[ link to this | view in chronology ]
Khaliq Musa Hadji, 10 Oct 2007 @ 10:27am

fs: Apple Iphone 4GB.....$250, Sony Ericsson P1...
KHALIQ PHONES INC.
We are specialised in all brand new mobile phones like Samsung , Nokia , Qtek , Imate , Palm Treo , Sony Ericsson , Sidekick , Nextel and many more, They are cheap and unlocked with 12 month international warranty.We sell them in Bulk and in Retails ,We shipped inconjunction with fedEx Express, they shipped within 48hrs to your door step. It is a Buy two and get one for free.

if interested contact : khaliq900@inmail24.com

Apple Iphone 4GB.....$250USD
Apple Iphone 8GB.....$350USD

PRICELIST FOR NOKIA PHONES IN STOCK :
Nokia 6125 Bluetooth Quadband Unlocked GSM Phone....$150
Nokia 6136 Bluetooth Quadband Unlocked GSM Phone....$160
Nokia 6265i Unlocked GSM Cell Phone.....$140
Nokia 6282 Unlocked GSM Smartphone......$170
Nokia 8800 GSM Unlocked GSM 900/1800/1900 World Phone.....$175
Nokia 770 Internet Tablet Support.....$165
Nokia 8801 GSM Unlocked GSM 850/1800/1900.....$180
Nokia E51 QuadBand Unlocked GSM Phone......$160
Nokia E61 Smartphone Unlocked GSM Phone......$200
Nokia E70 Unlocked GSM Smartphone.....$150
Nokia N70 Unlocked GSM Smartphone.....$155
Nokia N71 Unlocked GSM Smartphone.....$165
Nokia N72 Unlocked GSM Phone Black....$175
Nokia N81 8GB Quadband Unlocked GSM Smartphone Black.....$200
Nokia 7900 Prism Unlocked GSM....$190
Nokia N91 GSM Unlocked Phone.....$200
Nokia N92 Unlocked GSM Smartphone....$220
Nokia N93 Unlocked GSM Phone.....$220
Nokia N93i Unlocked GSM Phone...$270
Nokia N95 8GB Unlocked GSM Phone.....$300
Nokia N75 Unlocked GSM Phones....$350
Nokia 8800 Sirocco Edition ..$230

PRICELIST FOR IMATE PHONES IN STOCK :
I-Mate JAM 128MB Unlocked GSM Phone....$190
I-Mate JAM GSM/GPRS Pocket PC Unlocked Phone....$180
I-Mate JAMin GSM/GPRS Pocket PC Unlocked Quadband Phone....$200
I-Mate JASJAR Quadband GSM Unlocked PDA Smartphone.....$300
I-Mate K-Jam Quad Band Unlocked GSM Smartphone.....$210
I-Mate Smartflip Unlocked Quadband Smartphone.....$220
I-Mate SP5 Unlocked GSM Smartphone.....$130
I-Mate SP5m Unlocked GSM Smartphone....$120

PRICELIST FOR MOTOROLA PHONES IN STOCK :
Motorola Mpx 300...........USD$300
Motorola A1200 Unlocked Quadband Phone Black.....$170
Motorola A910 Unlocked GSM Phone.....$130
Motorola E1070 Unlocked GSM Phone.....$135
Motorola L2 Candybar RAZR Unlocked GSM Phone....$110
Motorola Q Unlocked GSM Phone.....$230
Motorola Rokr E2 Unlocked GSM Triband Phone w/iTunes....$190
Motorola V1050 Quad Band Unlocked GSM Phone.....$145
Motorola V195 Unlocked GSM Phone.....$135
Motorola V3 (Gold) Edition Unlocked GSM Phone....$130
Motorola V3c RAZR for Verizon.....$135
Motorola V3i Dolce & Gabbana Unlocked Quadband GSM Phone.....$165
MOTOROLA V3x (PINK) Unlocked GSM Phone w/128 MB Card.....$140
Motorola W220 GSM Unlocked Cell Phone.....$180
Motorola Razr Maxx V6 Unlocked Cell Phone.....$220

PRICELIST FOR SAMSUNG PHONES :
Samsung P920 Unlocked GSM Cell Phone......$245
Samsung I830 Smartphone Unlocked GSM Cell Phone....$220
Samsung Serene SGH-E910 Unlocked GSM Phone.....$300
Samsung S401i Unlocked Triband GSM Cell Phone....$180
Samsung D870 Unlocked GSM Phone.....$200
Samsung Z710 Unlocked GSM Mobile Cell Phone....$230
Samsung p900 unlocked GSM Cell phone......$255
Samsung p860 unlocked GSM Cell phone......$235

AK Phones:
AK Mobile AK-Mini....$160
AK Mobile AK-S1....$170
AK Mobile AK900.....$220
AK Mobile DV2...$230

O2 XdPhones
O2 Xda Atom.............USD$300
O2 Xda Atom Exec...USD$310
O2 Xda Stealth..........USD$320

GIGABYTE Phones:
GIGABYTE G-Smart i350...USD$350
GIGABYTE Doraemon....USD$300
GIGABYTE g-Cam.......USD$180
GIGABYTE Keroro......USD$210
GIGABYTE Snoopy......USD$215

Linx Phones:
Linx M1168......USD$170
Linx M2688........USD$175

PRICELIST FOR SONY ERICSSON PHONES :
Sony Ericsson K310 Unlocked GSM Phone....$125
Sony Ericsson K510 Unlocked GSM PhonE....$135
Sony Ericsson K610 Unlocked GSM Phone....$145
Sony Ericsson K790 Unlocked GSM Phone....$180
Sony Ericsson K800 Unlocked GSM Phone....$170
Sony Ericsson M600 Unlocked GSM Phone....$155
Sony Ericsson P990i GSM Unlocked.........$330
Sony Ericsson W700i Unlocked GSM Phone...$130
Sony Ericsson W810 MP3 Quadband GSM Unlocked Phone....$135
Sony Ericsson W850 GSM Unlocked Cell Phone.....$220
Sony Ericsson W900i Unlocked GSM Phone.....$230
Sony Ericsson W950 Unlocked GSM Phone.....$240
Sony Ericsson Z710 Quadband GSM Unlocked Cell Phone.....$240
Sony Ericsson p910i unlocked GSM phone.....$165
Sony Ericsson P1 unlocked GSM phone.........$250

PRICELIST FOR PALM TREO PHONES :
Handsping Treo 650 GSM Quadband Unlocked Smartphone PDA....$120
Handsping Treo 700w GSM unlocked phone .....$140

PRICELIST FOR SIDEKICKS:
Sidekick ID......$140
Sidekick 3......$140

PRICELIST FOR NEXTEL
Nextel i930..$140
Nextel i830...$120

PRICELIST FOR QTEK PHONES :
Qtek 8100 Tri-Band Camera Music Unlocked GSM Phone White.....$110
QTEK 8300 GSM MOBILE SMARTPHONE.....$120
Qtek 8310 Unlocked Quadband GSM Phone....$130
Qtek 8500 Unlocked SmartPhone.....$125
Qtek 8600 Unlocked GSM SmartPhone....$200
Qtek 9000 3G Pocket PC Unlocked GSM Phone.....$250
Qtek 9100 Smartphone / Pocket PC Unlocked GSM Phone....$150
Qtek 9600 Unlocked GSM SmartPhone.....$250
Qtek G100 GPS PPC and Pharos Ostia GPS Software(US Maps)....$155
QTEK S200 GSM/GPRS Pocket PC Unlocked Quadband Phone.....$165

Apple MacBook Pro Intel Core Duo 1.83 GHz 15.4" Display Macintosh Notebook..$550
Apple MacBook Pro Intel Core Duo 2 GHz ( Dual-Core ) 15.4" Display....$560
Apple PowerBook G4 Motorola PowerPC G4 1 GHz 17" Display Macintosh Notebook..$350
Apple PowerBook G4 PowerPC G4 1.33 GHz 17" Display Macintosh Notebook..$400
Apple PowerBook G4 PowerPC G4 1.67 GHz 17" Display Macintosh Notebook...$530
Apple PowerBook G4 PowerPC G4 1.67 GHz 15.2" Display Macintosh Notebook...$500
Apple Pro 15 Intel Core Duo 2.16GHz 15.4" Display Macintosh Notebook...$600
Apple Pro Intel Core 2 Duo T7600 2.33GHz Macintosh Notebook.....$640

If interested contact : khaliq900@inmail24.com
[ link to this | view in chronology ]
ALPHA LOAN FIRM, 5 May 2009 @ 10:45am

LOAN OFFERS
The Alpha Finance and Loans (A.F.L), is a certified loan lender that offers
loan to individuals and co-operate bodies.
We offer loan of any kind at low interest rate of 3% and with no credit
check for all international borrowers.
Any interested Applicants should contact The application desk of the
A.F.L at this email: (alphaloanfirm@gmail.com) for full application
proceedings.

ALPHA FINANCE & LOANS, US.
[ link to this | view in chronology ]