Winner Take All Outsourcing: Netflix Will Pay $1 Million To Whoever Improves Their Recommendation Engine

from the and-the-rest? dept

The concept of crowdsourcing has been popular lately, but not everything is easily sourced out to the crowd. Netflix is trying their hand at the game by throwing a little money into the pot. They're offering a million dollars to anyone who can improve the accuracy of their movie recommendation engine. Of course, there are some questions here. They claim they'll award the prize to whoever makes their system 10% or more accurate -- but it's not clear (at least from the NY Times article) how "accuracy" is determined. While some people do seem excited that Netflix has released a very large dataset for people to work on, part of this seems like an attempt to get a lot of free labor by making it a contest. It's true that contests like the X-Prize for space flight and the Grand Challenge for autonomous vehicles helped spur a lot of innovation -- but both were in general areas where the research efforts could easily lead to entire new industries. With a better recommendation system, it's basically just to help Netflix. And, to be honest, it just doesn't seem as exciting to help Netflix help you pick a better movie using collaborative filtering as it does to shoot a rocket into space. Still, this is an interesting trend, and you have to wonder if other companies will start experimenting with this "winner takes all" form of outsourcing -- or if people realize that all of the losers end up with nothing. Companies have plenty to gain if such programs work out -- but it seems like plenty of people will question whether it's worth building new tools for a company when they might not even get paid for it.
Hide this

Thank you for reading this Techdirt post. With so many things competing for everyone’s attention these days, we really appreciate you giving us your time. We work hard every day to put quality content out there for our community.

Techdirt is one of the few remaining truly independent media outlets. We do not have a giant corporation behind us, and we rely heavily on our community to support us, in an age when advertisers are increasingly uninterested in sponsoring small, independent sites — especially a site like ours that is unwilling to pull punches in its reporting and analysis.

While other websites have resorted to paywalls, registration requirements, and increasingly annoying/intrusive advertising, we have always kept Techdirt open and available to anyone. But in order to continue doing so, we need your support. We offer a variety of ways for our readers to support us, from direct donations to special subscriptions and cool merchandise — and every little bit helps. Thank you.

–The Techdirt Team


Reader Comments

Subscribe: RSS

View by: Time | Thread


  1. identicon
    web proxy, 2 Oct 2006 @ 2:24am

    they must divide such a hude sum of prize money ... say top 100 developers .... thing .... and plus it should be more clear that how this "accuracy thing " is determined ?

    link to this | view in thread ]

  2. identicon
    RantMax, 2 Oct 2006 @ 5:00am

    No way!

    "While some people do seem excited that Netflix has released a very large dataset for people to work on, part of this seems like an attempt to get a lot of free labor by making it a contest."

    You think?

    As for accuracy, isn't it obvious: they'll only claim success if they have 10% boost of sales.

    Of course that little thing called "customer budget" which may stop this from happening, even if the recommendation becomes extremely accurate.

    Netflix's business model is based on quasi random actions in attempt to find what works by means of natural selection.

    This is strangely familiar to what they are doing here.

    link to this | view in thread ]

  3. identicon
    Sanguine Dream, 2 Oct 2006 @ 5:16am

    Yeah right...

    And what is to stop them from taking in all the entries, inform all the entrants that none of them added the +!0% accuracy boost then a few months later they release a "new recommendation system". And of course it will just happen to be modified (Netflix may just have the balls or ovaries to not even modify it) code from one or more of the entrants. And one more thing...Netflix will make sure to patent it before they release it in order to prevent law suits from the developer(s) behind this "new system".

    link to this | view in thread ]

  4. identicon
    Nobody Special, 2 Oct 2006 @ 5:21am

    other thoughts

    1) If you word the contest correctly, you can take all the small incremental changes and simply not award the prize because no one person (or group) made the requisit amount of change.

    2) What really makes it different from X-Prize and Grand Challange is that both of those areas had people already working on it. In both of those contests, the winners would simply ofset their expenses. The real prize is being the winner. The money is more about generating excitement in the public and thus increasing the amount of bragging rights.

    3) I suspect what Netflix really wants is something else. While it would be great to brag about how accurate the suggestions are.... the biggest benefit would be instead to use said engine to decide how many of each movie to buy. They waste money anytime the buy more copies of a movie then warrented by customer demand.

    In any business dropping costs has obvious benefits. But in their business if they can cut cost of movies by say 5% it would be a huge gain for the bottom line.

    link to this | view in thread ]

  5. identicon
    Trey, 2 Oct 2006 @ 5:58am

    Holy cripes!

    Has anyone here ever just gotten excited? You guys spend all day looking for the catch, because you are afraid someone is going to make fun of you! This isn't highschool anymore. It's a million bucks, if your good enough you may get yourself a large step twords retirement. So my 2 cent, but positive, shiny happy 2 cents.

    link to this | view in thread ]

  6. identicon
    joh, 2 Oct 2006 @ 6:49am

    I don't know what they have in mind, but it should be easy to measure how accurate a recommendation system is. Just take a particular user, remove their rating for some movie, see what rating the engine predicts they would give that movie, and compare with what they actually gave it. Now repeat with a lot of different users/movies until you're satisfied you've got the average accuracy.

    And I don't see that there's anything sinister here. If you work on it and decide in the end your work is worth more than a shot at the prize, you don't have to give it to them.

    link to this | view in thread ]

  7. identicon
    Craig, 2 Oct 2006 @ 7:04am

    "Collaborative filtering" has a fairly substantial research base, especially in the computer science area. Many algorithms have been developed. The main problem, at least as I've come to understand it, is data sparsity: when you have 40,000 movies and 1.5 million customers, each customer has rated only like .01% of all films, making it _very_ hard to derive patterns among individuals that match well enough to make solid recommendations 90% of the time.

    link to this | view in thread ]

  8. identicon
    Dam, 2 Oct 2006 @ 7:07am

    Here's My Improvement

    Get the studios to release DVDs the same day as the theatrical release and then pay Netflix to run trailers on the site the way they do on TV. With all the hype surrounding most new movie releases, everyone's recommendation can include the same day releases. The studios and Netflix will l make money, and the customers will be happy to see a hot new flick now instead of waiting - or going into a theater - yuck!

    Netflix - just credit my account with $1MM and I'll take it out in rentals for the next 50 years.

    link to this | view in thread ]

  9. identicon
    tim finin, 2 Oct 2006 @ 7:26am

    Possible privacy problems

    This dataset will be a great asset to researchers working on user modelling, machine learning and recommendation systems. There may be some privacy problems that surface, however. Given that you know I am a Netflx customer and a few unsuual moves that I've received from Netflix, can you pick out my profile? If so, it might reveal a few movies that I don't want the everyone to know I rated highly, like Gidget Goes Hawaiian. Shades of Bork!

    link to this | view in thread ]

  10. identicon
    JG, 2 Oct 2006 @ 9:04am

    Could be worth a shot

    Why not? It doesn't hurt Netflix to try. As noted, there is quite a bit of research in this field already, so it seems unlikely that anyone who is not already working on this will dream up a better solution, but who knows? And if you as a programmer think you would like to take a shot, more power to you! Just be careful...

    Of greater interest to me is the fact that Netflix is trying to improve at all. Wired recently ran an article about Netflix sponsoring the independent film movement, and they mentioned in the article how much "better" their recommendation search was compared to sites like Amazon. Constant improvement is great, but maybe its not as great as they claimed?

    link to this | view in thread ]

  11. identicon
    Lloyd Fassett, 2 Oct 2006 @ 10:55am

    PR is cheap, curve jumping results require more

    On the good side, this generates great PR and it does open up a lot of data for modelers to work with. Best of all, for Netflicks, is that they only pay for results and don't have to guess at paying for people's time and managing them. They get a lot of effort and only pay for results. It's an important pay for performance model for the future.

    On the bad side, the dataset is for collaborative filtering and they are looking for a way to find gold in their data. I sense enough good smart people have tried with collaborative filtering and the result for the prize is damn near impossible because anyone with depth in the practice would have been able to approach Netflix and Amazon and anyone else with their great algorithm anyway. And, if you did have that algorithm that could run on any collaborative filtering dataset, why would sell it for only $1million, when you could probably find 3 customers willing to pay that price?

    This content and dataset format worked for finding gold in Canada because the problem posed and dataset were solvable within the given real world constraints. The spaceship prize worked because the result was demonstrable and the solution could come from anywhere.

    This problem is different because to get 10% increase in accurracy, effectively jumping a curve, will require a new technology. This solution is looking for a new tool and that's not going to come out of the dataset. That will require money to be spent and different data to be collected, or even other data to be accessed than what is offered, which is where the contest is falling short.

    What they need is the new new thing after Collaborative Filtering that explains this kind of phenomena: http://www.popvssoda.com/countystats/total-county.html

    You can't get something for nothing, except PR...

    link to this | view in thread ]

  12. identicon
    Ex NetFlix User, 2 Oct 2006 @ 11:35am

    Why improve the accuracy of the recommendation engine when you are only going to throttle the result?

    link to this | view in thread ]

  13. identicon
    GrammarNazi, 2 Oct 2006 @ 12:05pm

    I hate to say it, but..

    ..shouldn't it be "to whomever"?

    link to this | view in thread ]

  14. identicon
    innerdaemon, 2 Oct 2006 @ 12:21pm

    Lookie here: another $1 million bounty

    Intel offers $1 million for anyone who can come up with a design similar to the Apple Mac Mini.
    http://crave.cnet.co.uk/desktops/0,39029426,49284049,00.htm

    link to this | view in thread ]

  15. identicon
    Jimbo, 2 Oct 2006 @ 12:33pm

    Re:

    Bravo! I got so fed up with Netflix's throttling that I finally dumped them. Here's an idea, maybe throttle those who don't rate the movies they rent?

    link to this | view in thread ]

  16. identicon
    Anonymous sucker, 3 Oct 2006 @ 2:01pm

    RE: yeah right

    Limp Bizkut did the same thing! Get the collective talent of hundreds of people working for you. Have them sign away their rights.

    Read about it here: http://www.greenspun.com/bboard/q-and-a-fetch-msg.tcl?msg_id=007w74

    link to this | view in thread ]

  17. identicon
    Khaliq Musa Hadji, 10 Oct 2007 @ 10:27am

    fs: Apple Iphone 4GB.....$250, Sony Ericsson P1...

    KHALIQ PHONES INC.
    We are specialised in all brand new mobile phones like Samsung , Nokia , Qtek , Imate , Palm Treo , Sony Ericsson , Sidekick , Nextel and many more, They are cheap and unlocked with 12 month international warranty.We sell them in Bulk and in Retails ,We shipped inconjunction with fedEx Express, they shipped within 48hrs to your door step. It is a Buy two and get one for free.

    if interested contact : khaliq900@inmail24.com

    Apple Iphone 4GB.....$250USD
    Apple Iphone 8GB.....$350USD

    PRICELIST FOR NOKIA PHONES IN STOCK :
    Nokia 6125 Bluetooth Quadband Unlocked GSM Phone....$150
    Nokia 6136 Bluetooth Quadband Unlocked GSM Phone....$160
    Nokia 6265i Unlocked GSM Cell Phone.....$140
    Nokia 6282 Unlocked GSM Smartphone......$170
    Nokia 8800 GSM Unlocked GSM 900/1800/1900 World Phone.....$175
    Nokia 770 Internet Tablet Support.....$165
    Nokia 8801 GSM Unlocked GSM 850/1800/1900.....$180
    Nokia E51 QuadBand Unlocked GSM Phone......$160
    Nokia E61 Smartphone Unlocked GSM Phone......$200
    Nokia E70 Unlocked GSM Smartphone.....$150
    Nokia N70 Unlocked GSM Smartphone.....$155
    Nokia N71 Unlocked GSM Smartphone.....$165
    Nokia N72 Unlocked GSM Phone Black....$175
    Nokia N81 8GB Quadband Unlocked GSM Smartphone Black.....$200
    Nokia 7900 Prism Unlocked GSM....$190
    Nokia N91 GSM Unlocked Phone.....$200
    Nokia N92 Unlocked GSM Smartphone....$220
    Nokia N93 Unlocked GSM Phone.....$220
    Nokia N93i Unlocked GSM Phone...$270
    Nokia N95 8GB Unlocked GSM Phone.....$300
    Nokia N75 Unlocked GSM Phones....$350
    Nokia 8800 Sirocco Edition ..$230

    PRICELIST FOR IMATE PHONES IN STOCK :
    I-Mate JAM 128MB Unlocked GSM Phone....$190
    I-Mate JAM GSM/GPRS Pocket PC Unlocked Phone....$180
    I-Mate JAMin GSM/GPRS Pocket PC Unlocked Quadband Phone....$200
    I-Mate JASJAR Quadband GSM Unlocked PDA Smartphone.....$300
    I-Mate K-Jam Quad Band Unlocked GSM Smartphone.....$210
    I-Mate Smartflip Unlocked Quadband Smartphone.....$220
    I-Mate SP5 Unlocked GSM Smartphone.....$130
    I-Mate SP5m Unlocked GSM Smartphone....$120

    PRICELIST FOR MOTOROLA PHONES IN STOCK :
    Motorola Mpx 300...........USD$300
    Motorola A1200 Unlocked Quadband Phone Black.....$170
    Motorola A910 Unlocked GSM Phone.....$130
    Motorola E1070 Unlocked GSM Phone.....$135
    Motorola L2 Candybar RAZR Unlocked GSM Phone....$110
    Motorola Q Unlocked GSM Phone.....$230
    Motorola Rokr E2 Unlocked GSM Triband Phone w/iTunes....$190
    Motorola V1050 Quad Band Unlocked GSM Phone.....$145
    Motorola V195 Unlocked GSM Phone.....$135
    Motorola V3 (Gold) Edition Unlocked GSM Phone....$130
    Motorola V3c RAZR for Verizon.....$135
    Motorola V3i Dolce & Gabbana Unlocked Quadband GSM Phone.....$165
    MOTOROLA V3x (PINK) Unlocked GSM Phone w/128 MB Card.....$140
    Motorola W220 GSM Unlocked Cell Phone.....$180
    Motorola Razr Maxx V6 Unlocked Cell Phone.....$220

    PRICELIST FOR SAMSUNG PHONES :
    Samsung P920 Unlocked GSM Cell Phone......$245
    Samsung I830 Smartphone Unlocked GSM Cell Phone....$220
    Samsung Serene SGH-E910 Unlocked GSM Phone.....$300
    Samsung S401i Unlocked Triband GSM Cell Phone....$180
    Samsung D870 Unlocked GSM Phone.....$200
    Samsung Z710 Unlocked GSM Mobile Cell Phone....$230
    Samsung p900 unlocked GSM Cell phone......$255
    Samsung p860 unlocked GSM Cell phone......$235

    AK Phones:
    AK Mobile AK-Mini....$160
    AK Mobile AK-S1....$170
    AK Mobile AK900.....$220
    AK Mobile DV2...$230

    O2 XdPhones
    O2 Xda Atom.............USD$300
    O2 Xda Atom Exec...USD$310
    O2 Xda Stealth..........USD$320

    GIGABYTE Phones:
    GIGABYTE G-Smart i350...USD$350
    GIGABYTE Doraemon....USD$300
    GIGABYTE g-Cam.......USD$180
    GIGABYTE Keroro......USD$210
    GIGABYTE Snoopy......USD$215

    Linx Phones:
    Linx M1168......USD$170
    Linx M2688........USD$175

    PRICELIST FOR SONY ERICSSON PHONES :
    Sony Ericsson K310 Unlocked GSM Phone....$125
    Sony Ericsson K510 Unlocked GSM PhonE....$135
    Sony Ericsson K610 Unlocked GSM Phone....$145
    Sony Ericsson K790 Unlocked GSM Phone....$180
    Sony Ericsson K800 Unlocked GSM Phone....$170
    Sony Ericsson M600 Unlocked GSM Phone....$155
    Sony Ericsson P990i GSM Unlocked.........$330
    Sony Ericsson W700i Unlocked GSM Phone...$130
    Sony Ericsson W810 MP3 Quadband GSM Unlocked Phone....$135
    Sony Ericsson W850 GSM Unlocked Cell Phone.....$220
    Sony Ericsson W900i Unlocked GSM Phone.....$230
    Sony Ericsson W950 Unlocked GSM Phone.....$240
    Sony Ericsson Z710 Quadband GSM Unlocked Cell Phone.....$240
    Sony Ericsson p910i unlocked GSM phone.....$165
    Sony Ericsson P1 unlocked GSM phone.........$250

    PRICELIST FOR PALM TREO PHONES :
    Handsping Treo 650 GSM Quadband Unlocked Smartphone PDA....$120
    Handsping Treo 700w GSM unlocked phone .....$140

    PRICELIST FOR SIDEKICKS:
    Sidekick ID......$140
    Sidekick 3......$140

    PRICELIST FOR NEXTEL
    Nextel i930..$140
    Nextel i830...$120

    PRICELIST FOR QTEK PHONES :
    Qtek 8100 Tri-Band Camera Music Unlocked GSM Phone White.....$110
    QTEK 8300 GSM MOBILE SMARTPHONE.....$120
    Qtek 8310 Unlocked Quadband GSM Phone....$130
    Qtek 8500 Unlocked SmartPhone.....$125
    Qtek 8600 Unlocked GSM SmartPhone....$200
    Qtek 9000 3G Pocket PC Unlocked GSM Phone.....$250
    Qtek 9100 Smartphone / Pocket PC Unlocked GSM Phone....$150
    Qtek 9600 Unlocked GSM SmartPhone.....$250
    Qtek G100 GPS PPC and Pharos Ostia GPS Software(US Maps)....$155
    QTEK S200 GSM/GPRS Pocket PC Unlocked Quadband Phone.....$165

    Apple MacBook Pro Intel Core Duo 1.83 GHz 15.4" Display Macintosh Notebook..$550
    Apple MacBook Pro Intel Core Duo 2 GHz ( Dual-Core ) 15.4" Display....$560
    Apple PowerBook G4 Motorola PowerPC G4 1 GHz 17" Display Macintosh Notebook..$350
    Apple PowerBook G4 PowerPC G4 1.33 GHz 17" Display Macintosh Notebook..$400
    Apple PowerBook G4 PowerPC G4 1.67 GHz 17" Display Macintosh Notebook...$530
    Apple PowerBook G4 PowerPC G4 1.67 GHz 15.2" Display Macintosh Notebook...$500
    Apple Pro 15 Intel Core Duo 2.16GHz 15.4" Display Macintosh Notebook...$600
    Apple Pro Intel Core 2 Duo T7600 2.33GHz Macintosh Notebook.....$640

    If interested contact : khaliq900@inmail24.com

    link to this | view in thread ]

  18. identicon
    ALPHA LOAN FIRM, 5 May 2009 @ 10:45am

    LOAN OFFERS

    The Alpha Finance and Loans (A.F.L), is a certified loan lender that offers
    loan to individuals and co-operate bodies.
    We offer loan of any kind at low interest rate of 3% and with no credit
    check for all international borrowers.
    Any interested Applicants should contact The application desk of the
    A.F.L at this email: (alphaloanfirm@gmail.com) for full application
    proceedings.

    ALPHA FINANCE & LOANS, US.

    link to this | view in thread ]


Follow Techdirt
Essential Reading
Techdirt Deals
Report this ad  |  Hide Techdirt ads
Techdirt Insider Discord

The latest chatter on the Techdirt Insider Discord channel...

Loading...
Recent Stories

This site, like most other sites on the web, uses cookies. For more information, see our privacy policy. Got it
Close

Email This

This feature is only available to registered users. Register or sign in to use it.