What The Netflix Prize Tells Us About Innovation, Collaboration, Info Sharing And Game Theory

from the fascinating! dept

Tue, Jul 28th 2009 1:46pm — Mike Masnick

While there's lots of attention being paid to the fact that some team has won the Netflix Prize (it probably won't be announced who until September), there's an interesting side story that's worth noting -- which is how important collaboration was in breaking through. Plenty of studies have shown that innovation happens much faster when you have the free and open sharing of information (rather than having it locked up, say, by patents), as that mixture of different approaches and ideas allows for breakthroughs to come much faster (in fact, studies have shown that much of the success in Silicon Valley came from the free sharing of info across companies as people rapidly moved around).

And, in fact, that's exactly what happened with the Netflix Prize. The first "team" to break the 10% finish line, BellKor, was actually a merger of a few separate teams, allowing them to combine different pieces of different approaches to actually leap ahead. So, rather than trying to hoard the idea for themselves to claim the entire prize themselves, they realized it was better to team up to make the real breakthrough.

But, then a second interesting thing happened. Since the rules allowed another 30 days for other teams to offer up solutions that beat the first one, a bunch of other teams realized that it was in their best interest to team up as well, in order to leap-frog the original team. So they created the aptly named Ensemble -- and, again, the merger of various teams and different approaches allowed them to jump forward. It's not clear who actually had the best solution (both teams claim they did), but it's nice to see yet another clear example of the value of collaboration in innovation, against the standard myth of the lone inventor having a "flash of genius." It's also interesting to see the game theory aspect of the "loser teams" recognizing that they had to team up in order to catch up with the leader in the space.

Thank you for reading this Techdirt post. With so many things competing for everyone’s attention these days, we really appreciate you giving us your time. We work hard every day to put quality content out there for our community.

Techdirt is one of the few remaining truly independent media outlets. We do not have a giant corporation behind us, and we rely heavily on our community to support us, in an age when advertisers are increasingly uninterested in sponsoring small, independent sites — especially a site like ours that is unwilling to pull punches in its reporting and analysis.

While other websites have resorted to paywalls, registration requirements, and increasingly annoying/intrusive advertising, we have always kept Techdirt open and available to anyone. But in order to continue doing so, we need your support. We offer a variety of ways for our readers to support us, from direct donations to special subscriptions and cool merchandise — and every little bit helps. Thank you.

–The Techdirt Team

Filed Under: collaboration, game theory, innovation, netflix prize
Companies: netflix

36 Comments

If you liked this post, you may also be interested in...

Reader Comments

Subscribe: RSS

View by: Time | Thread

Anonymous Coward, 28 Jul 2009 @ 2:11pm

"it's nice to see yet another clear example of the value of collaboration in innovation, against the standard myth of the lone inventor having a "flash of genius.""

This would be true if we were talking of going from zero to 100 on something. But in this case, they are taking a very, very good system and trying to make it 10% better, in a very complex system.

So even if it is 10% better, the original 90% was done by a very, very small group of people having "AH-HA!" moments.

Reset, try again, another false conclusion.
[ link to this | view in thread ]
:Lobo Santo (profile), 28 Jul 2009 @ 2:41pm

Re:
Hello Coward!

To state my view succinctly: "Put up or shut up."
(Or if you're a Wikipedian: "citation needed.")

Do you have any *Real* data concerning the orginating algorithms and history thereof by which Netflix does it's suggestion system?

If so, please, by all means, support your conclusion with evidence.
[ link to this | view in thread ]
MindParadox (profile), 28 Jul 2009 @ 2:43pm

actually, it's not a false conclusion, nowhere in the article does it assume going from 0 to 100%, as a matter of fact, it simply says "breaking through" and "cross the 10% finish line"

besides, you have the false conclusion of 0 to 100% with your statement that "the original 90% was done by a very, very small group of people having "AH-HA!" moments." by the simple fact that you assume (falsely i might add) that the team of people working at netflix on their system was very small (it wasn't, believe me) or that it was ONLY their team that came up with the entirety of the idea (again, it wasn't, trust me, or do the research :))

but hey, pop another quarter in, and you get 4 more tries!
[ link to this | view in thread ]
CrushU, 28 Jul 2009 @ 2:45pm

Re:
After all, the wheel was only 10% better, if that, from the squares we had been using before. I mean, really, cutting the corners off?

We had a very good system in our squares before that, so all the people that thought of the wheel really didn't improve it very much.
[ link to this | view in thread ]
Space Pirate, 28 Jul 2009 @ 2:47pm

Re:
Was netflix built by a single ah-ha person? And can you really justify the the apparently arbitrary distinction between a 'small' group and the rather unknown team sizes?

Ahaaa!
[ link to this | view in thread ]
DJ (profile), 28 Jul 2009 @ 2:57pm

Re: Re:
Well done! I think that is the singularly most crushing argument I've seen on this site.
[ link to this | view in thread ]
Anonymous Coward, 28 Jul 2009 @ 3:05pm

Re: Re:
Actually, it's pretty simple. If 100 people made 90%, and 100 people made the last 10%, which one was more effecient?

The developer of Cinematch is Stan Lanning. I think he may be the original "ah-ha!" guy on this one.

Plus most of this is older news, the contest started in 2006 and BellKor was spotted in 2008.

http://www.wired.com/techbiz/media/magazine/16-03/mf_netflix

information on Cinematch from 2002:

http://www.wired.com/wired/archive/10.12/netflix.html?pg=1&topic=&topic_set=
[ link to this | view in thread ]
PRMan, 28 Jul 2009 @ 3:10pm

Cool, can they work on Watch Instantly now?
It stalls 2-3 times per episode.

Maybe it's Time Warner's fault, but I don't know.
[ link to this | view in thread ]
Ri, 28 Jul 2009 @ 3:18pm

Patents
They DID as a matter of fact violate dozens of patents. Every functional piece of software does!
[ link to this | view in thread ]
DJ (profile), 28 Jul 2009 @ 3:19pm

Re: Re: Re:
"Actually, it's pretty simple. If 100 people made 90%, and 100 people made the last 10%, which one was more effecient?"

You're applying an efficiency quotient to innovation. Coming up with the platform/process/invention (whatever you wanna call it) in the first place may have been the "hard part", but that doesn't mean that improving upon it was any easier.
[ link to this | view in thread ]
DJ (profile), 28 Jul 2009 @ 3:20pm

Re: Patents
For my argument on that point, I shall simply use one of the most versatile words in the English language.

Dude?!
[ link to this | view in thread ]
Mark Blafkin (profile), 28 Jul 2009 @ 3:32pm

Interesting but ...
not sure this is really an argument against patents.

Yes, it is a reminder that 'flash of genius' moments are relatively rare and that most innovative products are the result of collaboration and teamwork. However, often the teamwork happens inside singular companies...leading to great leaps. Sometimes those leaps are not enough, and companies need to buy other firms (think iPod and Kindle) or work out collaborative development agreements or licensing deals with other firms. Nothing about patents prevent this kind of collaboration. In fact, they often make it EASIER not harder because rights are clearly defined during the negotiation for collaboration.

This NetFlix prize is a great story, but it makes a case against ignoring the possibility of collaboration with perceived competitors. It doesn't make a convincing case against patents. Just look at Dean Kamen. No one was put more emphasis on creative competitions that encourage collaboration/coopetition, yet he is one of the biggest backers of the patent system.
[ link to this | view in thread ]
Anonymous Coward, 28 Jul 2009 @ 3:33pm

Re: Re: Re: Re:
No, all I am saying is that Mike is trying to say that "ah-ha" single inventor stuff is a "standard myth" and that this is much more effecient. My feeling is that for capping off the last 10.09%, perhaps that is right. But the original idea came down to a single guy or a very small group having an 'ah-ha' moment, not a crowdsource suddenly suggesting something.

The last 10% is probably much harder work than the first 90, so you need more people to push it there. But to deny how the first 90% happened is sort of like denying that a band or artist (like Trent Reznor) has an advantage because of his position in the public's mind, built up over years in the traditional music business.

But hey, what do I know? I apparently need to (retake) econ 101. ;)
[ link to this | view in thread ]
Ryan, 28 Jul 2009 @ 3:37pm

Re: Re: Re:
Actually, it's pretty simple. If 100 people made 90%, and 100 people made the last 10%, which one was more effecient?

You have obviously never heard of experience curves or the law of diminishing returns...
[ link to this | view in thread ]
Ryan, 28 Jul 2009 @ 3:41pm

Re: Re: Re: Re: Re:
Did you read the article? Given the exact same problem, individual groups failed to achieve a solution. Once they collaborated, they were able to make significant advances. Nobody ever said one person or two people or five people or fifty people are incapable of advancing technology; however, the more people the better, and compartmentalizing individual advances prevents symbiosis.
[ link to this | view in thread ]
Matt Tate (profile), 28 Jul 2009 @ 3:45pm

Re: Re: Re: Re:
My thoughts exactly. I'm not a doctor, but I figure that it's pretty difficult to make a good system 10% better, or else Netflix wouldn't put up a giant prize to do it.
[ link to this | view in thread ]
DJ (profile), 28 Jul 2009 @ 3:53pm

Re: Re: Re: Re: Re:
Actually, you're the first one to mention "ah-ha" moments. Mike's article is just describing how collaboration furthered innovation. He's NOT (that means he's NOT) taking anything away from orignial programmers/inventors. It's more of a "hooray-for-collaborative-innovation" article than anything else.

Which means you're arguing about a secondary issue....
[ link to this | view in thread ]
DJ (profile), 28 Jul 2009 @ 4:00pm

Re: Interesting but ...
Maybe I'm mistaken, but I don't think it's patents as a whole, that people are against. Rather, it's the way patent law is being handled in corporate America.
[ link to this | view in thread ]
drkkgt, 28 Jul 2009 @ 4:00pm

Re: Cool, can they work on Watch Instantly now?
wow, i use it a lot in LA with charter and have only had it hiccup once - but then it was my network connection since I was browsing the web from my lapti at the same time, couldnt load google and then everything came back and netflix kept on going. maybe TW is mucking with you.
[ link to this | view in thread ]
Kazi, 28 Jul 2009 @ 4:14pm

Who says that they only had to do only 10%? maybe they had to redo the whole thing because the system as previously designed reached it's limitations.

It's possible (Though I have no idea) that the work could have been done from the ground up ... redesigned the whole thing instead of working with something that already optimized completely.
[ link to this | view in thread ]
washii (profile), 28 Jul 2009 @ 4:26pm

Re: Re: Patents
But it's not as fun to examine the versatility of the word like with the F-Word.
[ link to this | view in thread ]
Ryan D, 28 Jul 2009 @ 4:44pm

Oh well...
...since it was ONLY 10% they might as well have not even done it anyways, I mean, if other people already had the 90%, why even make the effort if it only gains 10%

/sarcasm
[ link to this | view in thread ]
DS, 28 Jul 2009 @ 4:50pm

Wait, so someone did something, WITHOUT the gov't spending (my) money to fund it?

Gasp!
[ link to this | view in thread ]
Lawrence D'Oliveiro, 28 Jul 2009 @ 4:50pm

Re: Interesting but ...
Mark Blafkin spouted:

Nothing about patents prevent this kind of collaboration. In fact, they often make it EASIER not harder because rights are clearly defined during the negotiation for collaboration.

Patents don’t give you rights you didn’t have before. All they do is take away rights.
[ link to this | view in thread ]
Mike Masnick (profile), 28 Jul 2009 @ 4:58pm

Re: Interesting but ...
Just look at Dean Kamen. No one was put more emphasis on creative competitions that encourage collaboration/coopetition, yet he is one of the biggest backers of the patent system.

And have you noticed the commercial success of his various projects? It's no surprise that his businesses have been mediocre when it comes to the commercial market. He's someone who believes in the "big bang" theories of innovation rather than recognizing the importance of it being an ongoing process. Those who recognize the ongoing process are the ones who succeed in the market.
[ link to this | view in thread ]
Derek Kerton (profile), 28 Jul 2009 @ 10:48pm

Re: Re: Re: Re:
Yes. He DOES need to take Ecoc 101 again.
[ link to this | view in thread ]
Derek Kerton (profile), 28 Jul 2009 @ 10:55pm

Re:
You make a good point.

Netflix didn't start from Zero. Collaborative Filtering had been researched and published out of some other source. Xerox PARC, and a woman PhD at MIT, if I remember correctly.

I'm sure Netflix wasn't so stupid as to start from square one. They, like all inventors today, stood on the shoulders of many giants.

So, lemme see, they started based on prior work and academic research, probably used competitive analysis, too, looking at companies like Amazon and Tivo. They did a great job taking the concept further. They then realized they needed more collaborative brainpower to make it to the goal. I think the case against "flashes of genius" is pretty well made in this example.
[ link to this | view in thread ]
Derek Kerton (profile), 28 Jul 2009 @ 10:58pm

Re: Re: Interesting but ...
Yep. If he hadn't kept it such a secret, he might have known better what the market reaction would be to the Segway (a.k.a. IT). He might have even been able to modify the project to deliver something with stronger demand.
[ link to this | view in thread ]
Mike Masnick (profile), 28 Jul 2009 @ 11:06pm

Re: Re: Re: Re: Re:
No, all I am saying is that Mike is trying to say that "ah-ha" single inventor stuff is a "standard myth" and that this is much more effecient. My feeling is that for capping off the last 10.09%, perhaps that is right. But the original idea came down to a single guy or a very small group having an 'ah-ha' moment, not a crowdsource suddenly suggesting something.

Once again, we've actually shown numerous bits of research that have shown that the "a-ha" moment is mostly a myth. Not entirely, of course -- it has happened. But nearly all innovation comes out of a general advancement in knowledge, done in collaboration with groups.

It's the difference between recognizing that innovation is an ongoing process and thinking it's a single step.
[ link to this | view in thread ]
Mike Masnick (profile), 29 Jul 2009 @ 1:23am

Re: Re:

Netflix didn't start from Zero. Collaborative Filtering had been researched and published out of some other source. Xerox PARC, and a woman PhD at MIT, if I remember correctly.

You're thinking of Pattie Maes, and yes, Netflix is in many ways indebted to her and many others when it came to collaborative filtering (Maes, too, built on the work of earlier pioneers in the space). The idea that some lone engineer at Netflix had a flash of genius is a myth.
[ link to this | view in thread ]
Ender, 29 Jul 2009 @ 2:04am

Re:
To paraphrase a famous physicist, you aren't right, you aren't event wrong.

First of all, try to be aware of how the NetFlix contest actually worked. None of the participants ever had access to the code behind CineMatch. So each and every one of the participating teams had to reinvent the 90% you claim to be an 'A-ha' moment before they could proceed to do something better. Because of this, it's quite likely that the original NetFlix algorithm and both of the new potential winners have little to nothing in common.

There was no 'A-ha' moment to speak of, not in the sense you understand it anyway. Look up 'collaborative filtering' and papers related to this subject as well as 'neural networks' and 'machine learning'. There are many ideas and concepts that build upon each other, and the stared in the early 90s. When did you say the NetFlix team had the 'A-ha' moment and invented it all?
[ link to this | view in thread ]
Anonymous Coward, 29 Jul 2009 @ 5:45am

Re: Re:
"?When did you say the NetFlix team had the 'A-ha' moment and invented it all?"

I don't suggest for a minute that they have an "a-ha!" moment and invented the whole thing from nothing. Heck, someone had to invent math first, right?

But there is a time between "Netflix had nothing" and "netflix had something" where the idea was formed, honed, and put into place. It wasn't an infinite number of monkey moment, it wasn't a worldwide collaberation between hundreds of people, it was a guy or a small ground of guys sitting around working on an idea.

The real trick isn't in the methods used, it's in the idea and the general implementation of the idea. If you get 100 people in a room and don't tell them what to do, just to "invent something good" and you will get probably as many ideas as people - they certainly won't come up with one single idea. However, if you give them a focused goal, like "make this 10% better", it can work. My point is the initial spark doesn't come by rubbing a large group of people together. The original idea is as likely to have from two guys scribbling something on a coffee shop napkin or something quietly taking a bath and going "damn, we could do something like this" and bringing it into the next development meeting.

Truth is, if you can get large groups of people to work on something for free (or for the chance at a prize), and it is cheaper than trying to do it yourself, then it is good business. Netflix did 90% of the work in house over the last few years, honing and refining, and now they will pay out $1,000,000 for the last 10% of the deal. In house, it might have taken them 10 more years and 10s of millions to accomplish. But we wouldn't be here without the first 90%.

The "a-ha" moment in the case isn't coming up with a theory (collaborative filtering), but rather in realizing and understanding ways that it could be used to better provide movies to users.
[ link to this | view in thread ]
Mark Blafkin (profile), 29 Jul 2009 @ 7:05am

Re: Re: Interesting but ...
@Mike

First, I don't think the commercial success is relevant to the point I'm trying to make at all. Kamen's FIRST competitions for robotics are designed to encourage cooperation in the pursuit of prizes. Designing competitions that incentivize cooperation is actually something he spends a lot of time doing. He does believe in 'ongoing processes' of invention and cooperation to achieve such goals, despite what you assert here.

Second, you are simply wrong about his commercial success being "mediocre." While the Segway has not revolutionized cities the way some suggested it would, he has built a fortune by inventing, producing, and licensing dozens of medical technologies that have not only been runaway commercial successes but have also won him humanitarian and technology awards around the world. You are right that his attempt to essentially go it alone on producing and marketing the Segway was a mistake, but that seems less because of a lack of innovation than a lack of understanding of the market.

@Lawrence

Perhaps you misunderstood me. I know that some argue that patents "take away rights" of the populace at large. The patent system is a bargain that is designed incentivize the exploration of useful arts and sciences. Whether that bargain is currently the topic of serious debate.

However, I do not believe there is any question that patents confer special rights and privileges onto those that hold them.
[ link to this | view in thread ]
The Cenobyte, 29 Jul 2009 @ 11:38am

Ok so no Ah-Ha
What about all the data that shows it really doesn't matter what you do to try and keep information private and in your control. You had a story just a few days ago that showed that progress in the macro form can be plotted easily and doesn't seem to change no matter what extra factors are involved. So I will ask again, if progress on the Marco level doesn't change with or without patents, free licences, etc, how can we conclude that patents are a problem?
[ link to this | view in thread ]
gvhgv, 15 Jul 2015 @ 12:15am

hgvgvhg hgvhgvhg hgvhgv hgvhgv
[ link to this | view in thread ]
tubemate video downloader, 28 Jan 2016 @ 9:37am

nice
nice post try and good
[ link to this | view in thread ]