Can We Come Up With A Better Way To Measure Innovation?
from the the-market? dept
For quite some time, we've pointed out that patents are not a proxy for how much innovation a company does. In fact, research has proven this pretty conclusively. And yet, because they're easily countable, the press and politicians love to use the number of patents as a proxy for how much innovation is happening. That leads to silly articles from folks who should know better, making statements like "Clearly, the global recession seriously hampered innovation in the United States" because fewer patent applications have been filed.But, that leads to a separate issue. If you aren't using patents as a proxy, then how do you measure innovation? Tim O'Reilly is asking for suggestions on a measure innovation metric:
How might we construct a metric that would reflect the transformative power of the web (no patents), Google (nowhere near as many as their innovations), Facebook (ditto), Amazon (ditto, despite the 1-click flap), Craigslist, Wikipedia, not to mention free software such as Linux, Apache, MySQL and friends, as well the upwelling of innovation in media, maker culture, robotics... you name it: all the areas where small companies create new value and don't have time, money or inclination to divert effort from innovation to patents?The problem is that I'm not sure there is any single metric that really works here -- especially when it comes to disruptive innovations. You could go with revenue, but one of the features of truly disruptive innovations is that they sometimes shrink direct markets (while greatly increasing the size of indirect markets). So that might not be very useful either. You could go with user adoption -- but that may be fleeting or possibly gamable.
Even with older successful technologies, I'm not sure these kinds of metrics would most accurately highlight how much their innovation meant. Telcos made lots of money, but much of the real innovative value of the telephone was what other businesses they eventually enabled. How do you calculate that?
The only real metric I can think of -- though I'm not sure how one would measure this accurately -- is how much you would have to pay customers to get them to stop using a certain innovation. If you went around and surveyed people, and figured out how much it would cost to get them to, say, stop using search engines or email or mobile phones or automobiles, you might be able to get a sense of the "value" of certain innovations. From there, as a baseline, you could potentially monitor the delta over time. Thus, as the iPod grows in "value," the value of a portable CD player would decrease. As mobile phone cameras got better, the value of portable cameras would decrease, etc. It would be a lot of work, but could give you a much better general sense of innovation and how it changes over time than any patents.
Thank you for reading this Techdirt post. With so many things competing for everyone’s attention these days, we really appreciate you giving us your time. We work hard every day to put quality content out there for our community.
Techdirt is one of the few remaining truly independent media outlets. We do not have a giant corporation behind us, and we rely heavily on our community to support us, in an age when advertisers are increasingly uninterested in sponsoring small, independent sites — especially a site like ours that is unwilling to pull punches in its reporting and analysis.
While other websites have resorted to paywalls, registration requirements, and increasingly annoying/intrusive advertising, we have always kept Techdirt open and available to anyone. But in order to continue doing so, we need your support. We offer a variety of ways for our readers to support us, from direct donations to special subscriptions and cool merchandise — and every little bit helps. Thank you.
–The Techdirt Team
Filed Under: innovation, measurement, patents, tim o'reilly
Reader Comments
Subscribe: RSS
View by: Time | Thread
Patent inflation
Yet another aspect to this topic is that the quest for an accurate measure of innovation may be a red herring, diverting attention away from the more important measures of success. We're coming out of a period of time of massive innovation. Doesn't it make sense that there might be a bit of a lull in real new ideas for a while and the better measure of a company's worth isn't how innovative they are -- no matter what the metric is -- but instead, how effective the company is in using existing ideas?
[ link to this | view in chronology ]
Innovation over time
[ link to this | view in chronology ]
innovation depreciation?
[ link to this | view in chronology ]
[ link to this | view in chronology ]
Disinterest Value
For example: "How much money would convince you to stop using your iPhone for a week?" versus "How much money would convince you to stop using your iPhone forever?"
Basic Micro Economics refers to the use/satisfaction you get out of something as 'utility.' To the best of my knowledge, there is not yet a good metric for utility.
A metric for innovation... that's a tough one.
[ link to this | view in chronology ]
Hardware/Software update sales
[ link to this | view in chronology ]
It's easy
Long term:
1 divided by the number of patents filed.
Basically, if more patents are filed now, then it will be harder to innovate in the future. Also, as noted in the article, the most innovative companies seem to file the fewest patents. If more patents are being filed, it means that companies are probably spending too much time filing patents and not enough time innovating.
Short term:
1 divided by the number of patent lawsuits filed.
As noted many times before, companies that can't innovate litigate. A rise in the number of lawsuits filed indicates the number of companies that have given up on their own innovation and who want to stifle innovation by others.
[ link to this | view in chronology ]
Re: It's easy
As for your other measure, that too seems to have weaknesses. Patents filed is not inversely proportional to innovation. Mike claimed (I could not read the paper the posts linked through to, it seems to have been removed) that patents are UNRELATED to innovation. If that is true, your inverse proportionality would provide a meaningless number.
[ link to this | view in chronology ]
General metric.
value created
__________________________
people * resources used
So in basic terms more innovation will either use fewer resources and/or produce more/better things. I added people in there but maybe people would just be part of the resources bit? Not sure.
I think it has to be a bit general because innovation is tricky. For example, the Wii was an innovative game system (probably not very interesting from a value perspective) but the hardware from that combined with some open source software created inexpensive interactive whiteboards that make a wide range of other industries more productive. So in that case we have innovation, but how many innovation points do you give to the Wii vs the software?
[ link to this | view in chronology ]
Innovation == volatility or changes in expectations?
For example, consider measured life expectancy, and a prediction market in future life expectancy. Rises in both the volatility and level of the futures market give some reading about how much innovation people believe is in progress and likely. Conditional contracts -- (life expectancy for a diagnosis of condition X in 2020 | single-payer health-care) -- might offer best-guess estimates of the innovation expected under certain paths of action.
[ link to this | view in chronology ]
[ link to this | view in chronology ]
Punk?
[ link to this | view in chronology ]
For example, say I Invented the printing press. What was the old method of making copies? It was by hand. How long did that take? Say it took an average of 20 minutes per page in order to get 99 percent accuracy. Lets assume the first printing press, in comparison, can do 10 pages a minute with 95 percent accuracy (presses often mess up pages, you must throw pages away and fix the problem, they have all sorts of issues).
How much more efficient is the new system than the old? Well, you could see how many pages each does at 100 percent accuracy. At 100 percent accuracy the printing press does 9.5 pages per minute. Humans, on the other hand, do less than 1/20 pages per minute at 100 percent accuracy. That's a 190 fold difference. So we have just innovated by 19,000 percent.
Of course this doesn't measure the fact that when a printing press makes a mistake, it usually messed up the entire page whereby we can easily identify the mistake and discard it hence preventing us from ever replicating the same mistake. Some human mistakes, on the other hand, are far more likely to make it through filtration, and there is a chance that such mistakes could be copied and create a domino effect of mistakes where you end up with a document that is much different than the original. Sure, if you keep on copying a page and then copying that copy, etc... you wind up with a distorted image even with a printing press, but another innovation is the fact that, with the printing press, there is hardly ever a need to copy copies of copies, unlike the case when humans used to write stuff down. Of course all of these effects can also be separately measured and quantified but I don't want to go through the thought process required to do so and demonstrate it. One can also quantify the cost in labor for making copies vs the cost of operating and maintaining a printing press and the compare the cost per page both in time and money.
Now lets say someone invents the Internet. We can, again, compare the Internet as an information distribution platform to the latest printing press (not the above slow one). Say the latest printing press made 200 pages per minute and it would take an hour to deliver it to everyone in a neighborhood and the cost in labor was $600. The Internet, on the other hand, can deliver it all in 1 minute to everyone and the cost of labor is $15 (someone needs to type the reports up and post them. Lets forget the cost of the Internet vs the cost of the printing press). Again, we can easily compare the cost per person being delivered a service and quantify it and compare the service time's and hence, in a sense, quantify innovation and compare which method of doing something is more efficient and by how much (ie: percentage wise).
[ link to this | view in chronology ]
Re:
So I suppose, from an economics perspective, the best way to "measure" innovation is to determine how much additional consumer + producer surplus is created by an innovation. So to measure how much a country innovated during a period of time one would have to add the consumer and producer surplus created by each individual innovation during that time to get the total additional consumer and producer surplus created by all innovations during that time.
[ link to this | view in chronology ]
Why bother measuring innovation?
In football / soccer, you don't win games on the amount of passes you made down the wings (a tactic), you win games based on the number of goals you scored (a simplistic reading but I'm sure you get my drift). The thing that should be measured is the effect of the innovation on your goals - did our sales increase? What's our market share now?
[ link to this | view in chronology ]
How about The Innovation Power Score?
[ link to this | view in chronology ]
Great Metric
www.stepbystepinnovation.com
[ link to this | view in chronology ]