Can We Come Up With A Better Way To Measure Innovation?

from the the-market? dept

Mon, Mar 29th 2010 9:24am — Mike Masnick

For quite some time, we've pointed out that patents are not a proxy for how much innovation a company does. In fact, research has proven this pretty conclusively. And yet, because they're easily countable, the press and politicians love to use the number of patents as a proxy for how much innovation is happening. That leads to silly articles from folks who should know better, making statements like "Clearly, the global recession seriously hampered innovation in the United States" because fewer patent applications have been filed.

But, that leads to a separate issue. If you aren't using patents as a proxy, then how do you measure innovation? Tim O'Reilly is asking for suggestions on a measure innovation metric:

How might we construct a metric that would reflect the transformative power of the web (no patents), Google (nowhere near as many as their innovations), Facebook (ditto), Amazon (ditto, despite the 1-click flap), Craigslist, Wikipedia, not to mention free software such as Linux, Apache, MySQL and friends, as well the upwelling of innovation in media, maker culture, robotics... you name it: all the areas where small companies create new value and don't have time, money or inclination to divert effort from innovation to patents?

The problem is that I'm not sure there is any single metric that really works here -- especially when it comes to disruptive innovations. You could go with revenue, but one of the features of truly disruptive innovations is that they sometimes shrink direct markets (while greatly increasing the size of indirect markets). So that might not be very useful either. You could go with user adoption -- but that may be fleeting or possibly gamable.

Even with older successful technologies, I'm not sure these kinds of metrics would most accurately highlight how much their innovation meant. Telcos made lots of money, but much of the real innovative value of the telephone was what other businesses they eventually enabled. How do you calculate that?

The only real metric I can think of -- though I'm not sure how one would measure this accurately -- is how much you would have to pay customers to get them to stop using a certain innovation. If you went around and surveyed people, and figured out how much it would cost to get them to, say, stop using search engines or email or mobile phones or automobiles, you might be able to get a sense of the "value" of certain innovations. From there, as a baseline, you could potentially monitor the delta over time. Thus, as the iPod grows in "value," the value of a portable CD player would decrease. As mobile phone cameras got better, the value of portable cameras would decrease, etc. It would be a lot of work, but could give you a much better general sense of innovation and how it changes over time than any patents.

Thank you for reading this Techdirt post. With so many things competing for everyone’s attention these days, we really appreciate you giving us your time. We work hard every day to put quality content out there for our community.

Techdirt is one of the few remaining truly independent media outlets. We do not have a giant corporation behind us, and we rely heavily on our community to support us, in an age when advertisers are increasingly uninterested in sponsoring small, independent sites — especially a site like ours that is unwilling to pull punches in its reporting and analysis.

While other websites have resorted to paywalls, registration requirements, and increasingly annoying/intrusive advertising, we have always kept Techdirt open and available to anyone. But in order to continue doing so, we need your support. We offer a variety of ways for our readers to support us, from direct donations to special subscriptions and cool merchandise — and every little bit helps. Thank you.

–The Techdirt Team

Filed Under: innovation, measurement, patents, tim o'reilly

17 Comments

If you liked this post, you may also be interested in...

Reader Comments

Subscribe: RSS

View by: Time | Thread

Hulser (profile), 29 Mar 2010 @ 10:00am

Patent inflation
Another way to look at it is that patents actually could be good way to measure innovation if there wasn't a "patent inflation" caused by giving out patents for so many non-innovative ideas. The problem as I see it is that the standards for what's innovative has become so low, the count of how many patents a company has is all but meaningless.

Yet another aspect to this topic is that the quest for an accurate measure of innovation may be a red herring, diverting attention away from the more important measures of success. We're coming out of a period of time of massive innovation. Doesn't it make sense that there might be a bit of a lull in real new ideas for a while and the better measure of a company's worth isn't how innovative they are -- no matter what the metric is -- but instead, how effective the company is in using existing ideas?
[ link to this | view in thread ]
Melanie, 29 Mar 2010 @ 10:09am

Innovation over time
Such a way of measuring innovation will always capture a decline in 'value' for innovations that spark lots of competition. As more and more smartphones compete with the iPhone (or other innovations compete with smartphones), people would demand a lot less to stop using any one of them. They could just take the bribe and switch to a competitor. Does that mean that the original innovation is worth less? How do you decide at what point in time to measure? After widespread adoption but before significant competition?
[ link to this | view in thread ]
Hosermage (profile), 29 Mar 2010 @ 10:25am

innovation depreciation?
I don't see how you can depreciate innovation as it gets old. I can see how an innovation may not reach its full potential unless you give it enough time to evolve, but depreciate older innovations? I disagree. Just because iPod came along, it shouldn't discount CD-players when it was popular. In fact, I think the value of innovation should be accrued over time, so something like the wheel will have tremendous value since we've been using it for so long and its value will only stop growing until we stop using/producing it.
[ link to this | view in thread ]
Hoeppner, 29 Mar 2010 @ 10:30am

I can't think of a proper number that would let me stop using search engines. Would be even worse if it included "search this site".
[ link to this | view in thread ]
:Lobo Santo (profile), 29 Mar 2010 @ 10:38am

Disinterest Value
Your proposed metric needs a time dimension.
For example: "How much money would convince you to stop using your iPhone for a week?" versus "How much money would convince you to stop using your iPhone forever?"

Basic Micro Economics refers to the use/satisfaction you get out of something as 'utility.' To the best of my knowledge, there is not yet a good metric for utility.

A metric for innovation... that's a tough one.
[ link to this | view in thread ]
mike42 (profile), 29 Mar 2010 @ 11:02am

Hardware/Software update sales
How about hardware/software update/upgrade sales? If the new stuff isn't innovative, you won't buy it. It's got to have some feature that you're willing to spend money on, or at least spend time to download, for it to be worth while. You could probably cut out new sales, as that may be people just trying out the platform. In any case, it would be much more reasonable than counting patents.
[ link to this | view in thread ]
fogbugzd (profile), 29 Mar 2010 @ 11:38am

It's easy
There is a short-term metric and a long-term metric that come to mind.

Long term:
1 divided by the number of patents filed.
Basically, if more patents are filed now, then it will be harder to innovate in the future. Also, as noted in the article, the most innovative companies seem to file the fewest patents. If more patents are being filed, it means that companies are probably spending too much time filing patents and not enough time innovating.

Short term:
1 divided by the number of patent lawsuits filed.
As noted many times before, companies that can't innovate litigate. A rise in the number of lawsuits filed indicates the number of companies that have given up on their own innovation and who want to stifle innovation by others.
[ link to this | view in thread ]
Steven (profile), 29 Mar 2010 @ 11:42am

General metric.
I can't think of a good way to measure the innovation of a given company, but a general metric I think would be something along the lines of:

value created
__________________________

people * resources used

So in basic terms more innovation will either use fewer resources and/or produce more/better things. I added people in there but maybe people would just be part of the resources bit? Not sure.
I think it has to be a bit general because innovation is tricky. For example, the Wii was an innovative game system (probably not very interesting from a value perspective) but the hardware from that combined with some open source software created inexpensive interactive whiteboards that make a wide range of other industries more productive. So in that case we have innovation, but how many innovation points do you give to the Wii vs the software?
[ link to this | view in thread ]
gojomo (profile), 29 Mar 2010 @ 12:14pm

Innovation == volatility or changes in expectations?
At some level, innovation means effective changes in how results are achieved. So let's say there is a futures or prediction market in future results. The volatility, or changes in predicted levels, over time could each be considered indicators of the kind and volume of innovation.

For example, consider measured life expectancy, and a prediction market in future life expectancy. Rises in both the volatility and level of the futures market give some reading about how much innovation people believe is in progress and likely. Conditional contracts -- (life expectancy for a diagnosis of condition X in 2020 | single-payer health-care) -- might offer best-guess estimates of the innovation expected under certain paths of action.
[ link to this | view in thread ]
Anonymous Coward, 29 Mar 2010 @ 1:11pm

Re: It's easy
Your short-term measure does not work for the vast majority of companies. There are only around 2,800 patent lawsuits filed per year. There are an estimated 13 million companies in the U.S. What that means is that your short term measure will be meaningless for 12,997,200 companies.

As for your other measure, that too seems to have weaknesses. Patents filed is not inversely proportional to innovation. Mike claimed (I could not read the paper the posts linked through to, it seems to have been removed) that patents are UNRELATED to innovation. If that is true, your inverse proportionality would provide a meaningless number.
[ link to this | view in thread ]
Anonymous Coward, 29 Mar 2010 @ 4:04pm

how about how many times masnick uses your name as an example of good business models?
[ link to this | view in thread ]
angry dudu, 29 Mar 2010 @ 4:40pm

Punk?
Punky? Punk!
[ link to this | view in thread ]
Anonymous Coward, 29 Mar 2010 @ 5:19pm

I don't know if there is a good way to measure innovation but the traditional method has always been a function of how much more efficient/effective a new method is at doing something when compared to the old method.

For example, say I Invented the printing press. What was the old method of making copies? It was by hand. How long did that take? Say it took an average of 20 minutes per page in order to get 99 percent accuracy. Lets assume the first printing press, in comparison, can do 10 pages a minute with 95 percent accuracy (presses often mess up pages, you must throw pages away and fix the problem, they have all sorts of issues).

How much more efficient is the new system than the old? Well, you could see how many pages each does at 100 percent accuracy. At 100 percent accuracy the printing press does 9.5 pages per minute. Humans, on the other hand, do less than 1/20 pages per minute at 100 percent accuracy. That's a 190 fold difference. So we have just innovated by 19,000 percent.

Of course this doesn't measure the fact that when a printing press makes a mistake, it usually messed up the entire page whereby we can easily identify the mistake and discard it hence preventing us from ever replicating the same mistake. Some human mistakes, on the other hand, are far more likely to make it through filtration, and there is a chance that such mistakes could be copied and create a domino effect of mistakes where you end up with a document that is much different than the original. Sure, if you keep on copying a page and then copying that copy, etc... you wind up with a distorted image even with a printing press, but another innovation is the fact that, with the printing press, there is hardly ever a need to copy copies of copies, unlike the case when humans used to write stuff down. Of course all of these effects can also be separately measured and quantified but I don't want to go through the thought process required to do so and demonstrate it. One can also quantify the cost in labor for making copies vs the cost of operating and maintaining a printing press and the compare the cost per page both in time and money.

Now lets say someone invents the Internet. We can, again, compare the Internet as an information distribution platform to the latest printing press (not the above slow one). Say the latest printing press made 200 pages per minute and it would take an hour to deliver it to everyone in a neighborhood and the cost in labor was $600. The Internet, on the other hand, can deliver it all in 1 minute to everyone and the cost of labor is $15 (someone needs to type the reports up and post them. Lets forget the cost of the Internet vs the cost of the printing press). Again, we can easily compare the cost per person being delivered a service and quantify it and compare the service time's and hence, in a sense, quantify innovation and compare which method of doing something is more efficient and by how much (ie: percentage wise).
[ link to this | view in thread ]
Anonymous Coward, 29 Mar 2010 @ 8:19pm

Re:
(To continue)

So I suppose, from an economics perspective, the best way to "measure" innovation is to determine how much additional consumer + producer surplus is created by an innovation. So to measure how much a country innovated during a period of time one would have to add the consumer and producer surplus created by each individual innovation during that time to get the total additional consumer and producer surplus created by all innovations during that time.
[ link to this | view in thread ]
Simon (profile), 30 Mar 2010 @ 2:54am

Why bother measuring innovation?
Innovation isn't a goal in itself, it is a tactic; a means to an end.
In football / soccer, you don't win games on the amount of passes you made down the wings (a tactic), you win games based on the number of goals you scored (a simplistic reading but I'm sure you get my drift). The thing that should be measured is the effect of the innovation on your goals - did our sales increase? What's our market share now?
[ link to this | view in thread ]
Nick Kinports, 5 Apr 2010 @ 3:25pm

How about The Innovation Power Score?
Seems to solve the issue you are presenting: http://www.maddockdouglas.com/innovation-power-score
[ link to this | view in thread ]
John Gabrick, 9 Apr 2010 @ 4:33am

Great Metric
I've been doing this for over a decade, and I have to say that this is one of the best ways I've seen. All of the current methods: revenue, cost savings, number of patents, etc. have too many other dependent variables to make them a true indicator of value. In this case, the metric is simply time and money. When people dig into their wallet and pay for something that is the true value--you don't have to look any farther than eBay to see how time and money really set value.

www.stepbystepinnovation.com
[ link to this | view in thread ]