Can We Have Some Metrics On How Effective Metrics Are?
from the seduced-by-numbers dept
There's just something about "metrics." I spent Monday at the Word of Mouth Marketing Association's Research Symposium, which was supposed to be all about learning how to track the impacts of word of mouth marketing attempts. Of course, that makes an implicit assumption: that you can, need or want to track those things in a simple manner. It seemed like the key point made by many of the attendees (mainly from big companies) is that they need to figure out how to translate "WoM" campaigns back into the common terminology and "metrics" they use in traditional marketing in order to convince CEOs and marketing VPs that word of mouth campaigns make sense. Metrics was the word of the day, with multiple people saying that without metrics, word of mouth marketing was effectively pointless. There's just one problem with this: metrics can often be worse than useless. Metrics are only as good as what they're measuring, and they're often measuring either the wrong thing, or something that can easily be gamed. A month ago, Joel Spolsky had a fantastic post on the ridiculousness of metrics, and how they're often used mainly by consulting firms to convince big companies to funnel money into the consultants' bank accounts.With all of that in mind, it's not that surprising to see a story noting that web analytics firms (who have been at this for quite some time, and you would have thought had the kinks worked out) are revising the traffic certain sites got downward, often by tremendous amounts. Apparently, the website Entrepreneur.com didn't have 7.6 million visitors (as reported), but just over 2 million. That's not a small error. The problem, it seems, is that the site put news content into unwanted pop-up ads, often delivered by adware on lots of computers. In other words, the pageview may have happened, but it wasn't wanted, and it's quite likely that it was closed before it even loaded (perhaps with anger). In other words, it was a metric that was gamed. Keep this in mind when you hear silly reports like MySpace passing Yahoo in pageviews according to various metrics companies. As the article linked here notes, part of that may be due to Yahoo shifting their web mail app to AJAX, making it much more efficient and user friendly -- but requiring fewer page views. Equally responsible, however, would be MySpace's awful design that requires extra page impressions to actually get anything done. In other words, in focusing so much on the "page views" metric, the incentives encourage bad design. The companies that win the metrics game are the ones who are designed in ways that upset users, rather than those that make their service better, faster and more efficient. Normally, that would be a clear indication that the metrics are all wrong -- but very few people seem to care. They just continue focusing who is ahead of whom on the list -- and plenty of bad decisions continue to get made because of this blind allegiance to metrics.
Thank you for reading this Techdirt post. With so many things competing for everyone’s attention these days, we really appreciate you giving us your time. We work hard every day to put quality content out there for our community.
Techdirt is one of the few remaining truly independent media outlets. We do not have a giant corporation behind us, and we rely heavily on our community to support us, in an age when advertisers are increasingly uninterested in sponsoring small, independent sites — especially a site like ours that is unwilling to pull punches in its reporting and analysis.
While other websites have resorted to paywalls, registration requirements, and increasingly annoying/intrusive advertising, we have always kept Techdirt open and available to anyone. But in order to continue doing so, we need your support. We offer a variety of ways for our readers to support us, from direct donations to special subscriptions and cool merchandise — and every little bit helps. Thank you.
–The Techdirt Team
Reader Comments
Subscribe: RSS
View by: Time | Thread
typo
Typo in the top section.
'metrics can often worse than useless'
[ link to this | view in chronology ]
Re: typo
Oops. Fixed. Thanks.
[ link to this | view in chronology ]
Measurable statistics will always outweigh intelligence, design, and accessibility.
Nice post with good articles/sources cited... however, again, this isn't newsworthy... just common sense and something that is to be expected.
[ link to this | view in chronology ]
Metric
When in fact it turned out that consultants have just latched onto a new word, remember when the word Paradigm was so popular a few years ago? Or when people started adding e and z's to words that had no business having them?
It was stupid. Still is.
[ link to this | view in chronology ]
duh!
Yahoo gets paid per click, thus they want to minimize page loads (no sense wasting extra bandwith.. )
The only metric I care about lately is the number of unique visitors I get. If it's growing, things are good. If it's shrinking, things are bad.
The other good metric is activity. what percent of unique visitors post comments, click ads, buy stuff, or "do" whatever there is to do on the website.
[ link to this | view in chronology ]
Metrics v. faith
The accountants and business school types are taught that if something can't be measured it can't be managed. But WoM isn't completely manageable. If it's going to work, it has to take on a life of its own. That's really what blogging and social media are all about -- getting away from the management and manipulation in order to encourage real conversations betwen real people.
Much of marketing is like going to church. If you believe going to church has value for you then it does. But you can't measure that value or put it in a spreadsheet. The more people try to measure the value of most types of marketing (direct marketing excluded), the less they'll focus on actually delivering anything of value.
[ link to this | view in chronology ]
Gaming the Metric
The reality is that he can only measure page views, and estimate how page views convert to sales (either by direct click-through or later visits to the sales site). So we measure and pay on page views, and site operators have the goal of increased views, no matter how bad the content they have to build to get them.
Metrics are good only insofar as they measure the thing we are really interested in.
[ link to this | view in chronology ]
Measurinfg metrics...
I'm sure Google can come up with excellent metrics on something or another, and people liove them. Perhaps you were referring to companies that rely solely on pageviews as a metric?
For the rest of it, I think you are absolutely right that metrics can be manipulated and WoM is hard to quantify, but the nature of business requires some attempt at quanitification. Social networking etc is great for meeting people and stuff but until a reliable way to equate the effectiveness of social networking and WoM to predictable consumer patterns (or opinions) it's a tool that most businesses will not invest too heavily in.
Can you imagine some marketing guy going to the CEO of a company and saying "Hey, our product is going to sell lots of units because a lot of people are blogging about it and response has been favorable." The first questions the CEO is going to ask are 1) How many people are blogging about it and 2) What does favorable response mean in terms of predicted sales? Based on that answer money is allocated, product numbers are set, and schedules are made.
If the vast difference between the WoM generated by "Snakes On A Plane" versus the actual ticket sales is any indication, corporations are going to be very shy about utilitizing WoM as a metric.
[ link to this | view in chronology ]
acronyms.
[ link to this | view in chronology ]
Good metrics are difficult
Gaming of metrics will occur, and it is up to the consumer of the metric to critically view the information presented by the metric and interpret how to best respond.
The odometer of a used car for sale is an example of a metric. You can use it to estimate the level of use/abuse the car has taken. The metric can be "gamed" by shady dealerships, although development by auto manufacturers is making this more difficult, and services like carfax can help to support whether the metric is accurate.
Creating good metrics is a difficult activity, and presenting the data calculated by the metrics is also somewhat of an art. Joel Best's book, Damned Lies and Statistics, addresses this topic better than I can.
Saying metrics are "rediculous" is short sighted. Joel Spolsky's article addresses the difficulties in using metrics to manage software developers. While this may be true for that industry (and I'd argue he's using metrics that are too broad and subject to gaming), it's not universally true.
Metrics can be useful. Consider calories per serving, miles per gallon, P/E ratios, and processor clock speed. All may help you in making a purchasing decision - it's a matter of how you use or ignore them.
Metrics are numbers. Numbers are neutral. The creation of the metrics and (ab)use of them is where the trouble begins. But I repeat myself.
And as for ignoring metrics, consider the risks next time you have a pair of flashing red or blue lights behind you as you are driving the highway. Was that "miles per hour" metric-producing speedometer useful or "rediculous".
[ link to this | view in chronology ]
all metrics are equal (yeah right) but some are mo
[ link to this | view in chronology ]
Metrics indeed
Great thoughts. I agree that metrics are needed but that closer scrutiny is needed, after all, statics of any kind can (and are) manipulated all the time. It'll be interesting to see how improvements such as AJAX-enabled functionality (and the advent and implementation of Adobe Apollo-style rich internet applications) will impact web metrics.
We're certain to see big changes in the way we measure (and what we measure) in the future.
Thanks for thinking out loud about the metrics issue.
Scott Abel TheContentWrangler.com
[ link to this | view in chronology ]
comment on article
[ link to this | view in chronology ]