Of Clotheslines, Black Swans And Bad Measurements
from the we-measure-the-wrong-things-and-we-do-so-badly dept
Back in April, in writing about Union Square Ventures' Hacking Society event, I discussed the importance of measuring the unmeasurable, in noting that we all too often seem to be evaluating information-era economics using industrial-era metrics. That's a problem. Nick Grossman, who organized that Hacking Society session, has a great post discussing this same concept and highlighting Tim O'Reilly's discussion about this topic, in which he describes the clothesline paradox, which actually seems to come from a discussion in the early 1970s, and highlights how metrics can mislead. You can think of the clothesline paradox like this:If you take down your clothes line and buy an electric clothes dryer the electric consumption of the nation rises slightly. If you go in the other direction and remove the electric clothes dryer and install a clothesline the consumption of electricity drops slightly, but there is no credit given anywhere on the charts and graphs to solar energy which is now drying the clothes.In my mind, there are two "problems" associated with this, and while I think there is interest in attacking the first one, the second problem is often ignored. The first problem is that we notice that important information is measured with the wrong metrics. We see this all the time in the internet era. People talk about "the collapse" of the music industry, but miss the fact that more music has been produced, recorded and released in the last decade than in any previous decade. In fact, some of the evidence suggests more music was produced and recorded in the last decade than all other decades combined. Of course, that's an example of a metric that can be determined, but not all such metrics are that easy to pin down. For example, we talked about how Craigslist almost certainly helped contribute to the challenge that many newspapers are facing, because it undercut the cash cow that supported many of them: the classified advertising business. And if you used traditional metrics, you'd bizarrely and incorrectly suggest that Craigslist somehow "destroyed" value. But that's because no one takes into account all the value that Craigslist created, not for itself, but for its users. But how do you measure the fact that I can now find someone to take my old couch away for free? There's value in that transaction, but no one "measures" it. What about the fact that I can more efficiently rent out an apartment - without having to pay the local newspaper? Again, there's value, but it's not properly measured.
The second problem is a little trickier to understand. It's that when we have things that we can measure, we instinctively gravitate towards using those metrics, even if they're the wrong metrics! I was thinking about this as I read Paul Graham's excellent thoughts on "black swan farming," which is all about the counter-intuitive process involved in funding startups. There's a ton of tremendously thought-provoking lines in that piece, but I'm going to concentrate on one, which was really more of an aside, unrelated to the larger article (which you should go read), because it helped clarify my thinking on this point. Graham talks about not bothering to measure how many of the YCombinator companies he funds and trains later go on to raise more money after their initial fundraising efforts, noting:
I deliberately avoid calculating that number, because if you start measuring something you start optimizing it, and I know it's the wrong thing to optimize.And here's where the problem of using the wrong metrics becomes compounded. Even if you know something is the wrong metric, just having the number almost forces you to optimize for it. So rather than looking at, say, what's best for the overall culture of music, we look at "revenue for the record labels" and decide we need to "fix" that. Or, we look at the patent system as a proxy number for "innovation" and then the focus becomes solely on increasing the number of patents we issue, rather than on actually maximizing innovation.
When you have the wrong metrics, not only do you have bad or incomplete information, but even when you know that it's almost impossible not to optimize for those metrics, because you don't have anything else to work towards.
There is a lot of new interest in quantifying all sorts of new data -- and one benefit of the information age is that it also helps to create new data that can be quantified. But not all quantified data is actually that useful, and unfortunately, we often get so focused on the fact that we have a number, we ignore the possibility that the number is not telling us anything useful.
I was recently reminded of Shelby Bonnie's opinion piece from three years ago about why we need to kill the CPM as a metric for advertising (for those who don't know, CPM -- or "cost per thousand" impressions -- is how most banner ads are sold). He noted, quite accurately, that even those with the best of intentions to get away from "CPM-based" advertising seem to end up there in the end anyway. Because we have that number. And it becomes what people optimize around, just because it's there.
All campaigns start with the best of intentions: “let’s do something creative, engaging, and unique!” But unless someone really senior from the agency or client side intervenes, the road for a campaign always leads to the media buyer and the dreaded spreadsheet, where the two most important columns are impressions and cost. Ironically, there’s usually some good stuff in campaigns, but they are thrown in for free as “value adds.” At some point, publishers decide that if all clients care about is impressions, then OK, we’ll give them impressions. The output is an industry that overproduces shallow, superficial, commoditized impressions. Why do we have so many bad sites that republish the same junky content–content that’s often made by machines or $1-per-post contractors? Why do sites intentionally try to get us to turn lots of pages with tons of top 10 lists, photo galleries, or single-paragraph summaries of someone else’s story?The more I spend time thinking about these issues, the more I think these combined problems -- both not having the right data and then optimizing for the wrong data -- are the keys to many of the issues that we're regularly discussing around here. Figuring out ways to get beyond that, and to find the right data, and break our habits of relying on bad data are going to be increasingly important.
Thank you for reading this Techdirt post. With so many things competing for everyone’s attention these days, we really appreciate you giving us your time. We work hard every day to put quality content out there for our community.
Techdirt is one of the few remaining truly independent media outlets. We do not have a giant corporation behind us, and we rely heavily on our community to support us, in an age when advertisers are increasingly uninterested in sponsoring small, independent sites — especially a site like ours that is unwilling to pull punches in its reporting and analysis.
While other websites have resorted to paywalls, registration requirements, and increasingly annoying/intrusive advertising, we have always kept Techdirt open and available to anyone. But in order to continue doing so, we need your support. We offer a variety of ways for our readers to support us, from direct donations to special subscriptions and cool merchandise — and every little bit helps. Thank you.
–The Techdirt Team
Filed Under: data, economics, metrics, optimization, studies
Reader Comments
Subscribe: RSS
View by: Time | Thread
[ link to this | view in chronology ]
Re:
[ link to this | view in chronology ]
Re: Re:
[ link to this | view in chronology ]
Re: Re: Re:
And you're now reading this in my voice! Oh, my, yes.
[ link to this | view in chronology ]
Sometimes Reality gives too Much.
Maybe people in charge needed a few more of those in school.
[ link to this | view in chronology ]
Re: Sometimes Reality gives too Much.
Finding and combining them in the right way is one of the highest levels of scoring in terms of demonstrating your knowledge in a field on universities!
It is the hardest lesson to learn and since reality often is too complex for optimizing or your measurements are too bad, you almost always have to settle for some sub-optimal measures.
Modellers dilemma: Too many variables makes the demand of accuracy on your measurements too high. Too few variables and your model will wilther away in the universe of: Just another mediocre correlation.
[ link to this | view in chronology ]
[ link to this | view in chronology ]
[ link to this | view in chronology ]
The problem with economics as a whole
Someone just posted this on Facebook and it seems relevant here. From a speech over 40 years ago.
Robert F. Kennedy challenges Gross Domestic Product - YouTube
[ link to this | view in chronology ]
Re: The problem with economics as a whole
Thanks Suzanne. A simple statement like that seems to sum up why Mike always seems to be making odd claims, because economics tends to measure the wrong stuff.
Measuring marginal costs when the majority of the expensive is up front: WRONG.
Using a lack of sales as a justification for piracy not hurting anyone: WRONG.
Thinking that short term paint color marginal improvements on a product is true innovation: WRONG.
Thanks for stating in simple terms why Mike seems to be wrong very often. I also thank Mike for this big long post, as it clearly explains why he is wrong - because he looks in the wrong places for his data, and ignores the bigger picture.
[ link to this | view in chronology ]
Re: Re: The problem with economics as a whole
Ugh. We don't measure marginal cost while ignoring total cost. That's a strawman you made up. We merely point out -- accurately -- that marginal cost is what matters in pricing. Total cost does matter. It matters in making the go/no go call for the investment, something we've stated over and over again.
You're simply making stuff up when you claim that we ignore total costs for marginal costs. And you display an ignorance of economics when you claim that "measuring marginal cost" is somehow "wrong."
That's not a "metrics" problem as discussed here. If you think that, you're even more ignorant of economics than I thought.
[ link to this | view in chronology ]
Re: Re: Re: The problem with economics as a whole
yet, plenty of economists don't agree with you, most of them pretty much suggest that you shouldn't use the marginal cost model in a high up front cost business, as it's misleading. The marginal costs don't make it possible to set an acceptable price in the market place.
But hey, if you want to intentionally ignore what you learned in school, that's up to you!
[ link to this | view in chronology ]
Re: Re: Re: Re: The problem with economics as a whole
Yeah, so let's pass some laws that guarantee profit for old school industry. Too big to fail? I dont think so.
[ link to this | view in chronology ]
Re: Re: Re: Re: The problem with economics as a whole
I have never seen an economist suggest this. Perhaps you should go back to school. Once again, you seem confused about what is meant by marginal cost. But, you have been confused about that topic going on five or six years now, so at some point I have to conclude the problem is just you.
[ link to this | view in chronology ]
Re: Re: Re: Re: Re: The problem with economics as a whole
Quite simply, if the cost to produce the first unit is so far beyond what the market will bear, then the pure marginal costs of the next unit don't even begin to address the realistic pricing of the product.
Let's take a simple example: Someone creates a new movie. It cost them $1 million to make it, distribute it, everything. Now, in your "infinite" universe, the marginal cost is effectively zero. The problem? The price doesn't address the million spent up front. That fixed cost, while not part of your magical marginal cost arguments, means that pricing based only on marginal cost will fail, because it will drive producers out of the market place.
Remember: When you reach zero marginal cost, you also reach zero marginal benefit. Without a benefit, nobody comes to the table.
You could read stuff like:
http://pages.stern.nyu.edu/~wbaumol/OptimalDeparturesFromMarginalCostPricing.pdf
or
http://www.stanford.edu/~rehall/Relation-Price-JPE-Oct-1988.pdf (a little more off the beaten path, this one is a harder read).
Further, you can look around the internet and discover a number of papers discussing how infinite distribution is in fact not infinite, as you quickly reach the point where you either run out of population (they are limited) or run out of people who are interested in your product. You will never have true infinite distribution, because you will never produce an infinite number of copies.
Finally, in discussions of marginal costs and marginal cost pricing, almost all economists point to total average cost as the true indication of the price needed to sell the product in order for companies to remain in business (it's the break even number). Since we have already shown that your distribution is not infinite (there are a finite number of consumers for any given digital product), therefore total costs can in fact be calculated on a per item basis.
There is an interesting twist here of course: Since the market for digital goods is finite (for each product), piracy does have the effect of shrinking the potential market. So when it comes to total cost calculation for the break even point for selling the product, with the smaller market more of the total cost must be assigned to each remaining potential sale. in simple terms, if you shrink the market enough (so there is only 1 single movie ticket buyer left) you would have to sell the ticket for 1 million, even if you have infinite distribution and no marginal costs - because the marginal costs are NOT RELEVANT ANYMORE.
You really should spend more time investigating this Mike. The economy of abundance basically parks the marginal costs arguments out of the discussion, because they are no longer relevant to making a business work. If anything, it looks like pricing using things like price discrimination may be the long term way of dealing with the digital goods market.
Perhaps one day we can meet up and we can look at each other's papers on the subject. School was fun!
[ link to this | view in chronology ]
Imagine
We measure unemployment by counting how many folks applied for unemployment in the last six(?) weeks. To me, unemployment is how many folks can work vs how many folks are working.
Now those two examples do not equate to what Mike is talking about directly. But they are examples of how political forces (not just government) work to obfuscate real issues.
Think Hollywood Accounting!
[ link to this | view in chronology ]
Re: Imagine
Or how people are required to work to produce our current everything vs how many are actually working. I suppose the negative number would confuse even further, though.
[ link to this | view in chronology ]
Re: Imagine
Imagine someone telling you that the foundation and structure wasn't important, but how you delivered the keys to the clients was absolutely critical. That would be Mike focusing in on marginal costs while dealing with large up front cost business structures. You could make a business case out of it, perhaps, but you would be truly missing most of the deal.
[ link to this | view in chronology ]
Re: Re: Imagine
For the most part, we're not arguing against that: we're arguing against unsustainable business models. Selling digital content and only digital content? Not sustainable. Selling digital content as part of a larger bundle? Sure. Selling t-shirts, watches and general merchandise surrounding the content? Sure.
Just because a total cost exists does NOT mean that we must buy at that price or else. What it means is that, for your product to be viable, people must buy at that price. That doesn't apply to technically infinite goods (such as .mp3s and .avis). To try and make the world conform to you is akin to The Folly of Cnut.
[ link to this | view in chronology ]
Re: Re: Re: Imagine
It's a fail. If people don't want the "stuff", your content is worthless because you are whoring it out trying to hock stuff.
It's just not very good business sense.
[ link to this | view in chronology ]
Re: Re: Re: Re: Imagine
Glad you are in agreement at last.
[ link to this | view in chronology ]
Re: Re: Re: Re: Re: Imagine
So I am glad you agree. Can you explain this to Mike and Marcus now?
[ link to this | view in chronology ]
Re: Re: Re: Re: Re: Re: Imagine
lol
[ link to this | view in chronology ]
It's a case where the issue of piracy is a market issue not a product issue and by using metrics produced at the product level you are distorting your reality of the over all market. In the case of iOS we can show this in how piracy happens on jailbroken phones and that is a metric that is measured (10-20% when I last looked) yet app devs talk about a 90-95% piracy rate. How you deal with a market is vastly different when you are looking at most 20% of people who might be pirates then 95% of people.
[ link to this | view in chronology ]
Your music industry example is right on spot. You simply MUST add shows, merchandise and other forms of making money that actually have nothing to do with selling CDs or mp3. Heck, I'd say you need to include crowdsourcing efforts, small groups presenting on bars and many other sources of revenue before stating anything about the music business. The RIAA is but a small part of the pie and the fact that this slice is shrinking steadily doesn't mean the pie isn't growing fast.
[ link to this | view in chronology ]
Software development
It kills me a little every time someone starts talking about SLOC...
[ link to this | view in chronology ]
I read the whole thing including "farming black swans" article
[ link to this | view in chronology ]