You, Yes You, Are To Blame For Junk Stats

from the 9-out-of-10-people-disagree dept

Thu, Jun 4th 2009 9:23pm — Carlo Longino

It's pretty easy to find examples of bad statistics, whether they stem from poor data collection or interpretation. But a writer over at the BBC thinks he's fingered the real culprit: people who respond to surveys incorrectly. While on one level, his claim may have a little bit of truth to it, it really seems like an abdication of responsibility for the media, who all too often don't check out the bogus stats they cite, or swallow manipulated data without much question. Furthermore, when many groups use push polls not just to collect data, but to influence opinions, it's hard to blame the average respondent too much. In short, bad responses to survey questions could be a problem -- but that doesn't excuse the people collecting the data, and especially the media reporting it, from ignoring the issue. If they know the data's inaccurate, why do they keep reporting it?

Thank you for reading this Techdirt post. With so many things competing for everyone’s attention these days, we really appreciate you giving us your time. We work hard every day to put quality content out there for our community.

Techdirt is one of the few remaining truly independent media outlets. We do not have a giant corporation behind us, and we rely heavily on our community to support us, in an age when advertisers are increasingly uninterested in sponsoring small, independent sites — especially a site like ours that is unwilling to pull punches in its reporting and analysis.

While other websites have resorted to paywalls, registration requirements, and increasingly annoying/intrusive advertising, we have always kept Techdirt open and available to anyone. But in order to continue doing so, we need your support. We offer a variety of ways for our readers to support us, from direct donations to special subscriptions and cool merchandise — and every little bit helps. Thank you.

–The Techdirt Team

Filed Under: junk stats, stats

12 Comments

If you liked this post, you may also be interested in...

Reader Comments

Subscribe: RSS

View by: Time | Thread

lovingseem (profile), 4 Jun 2009 @ 9:31pm

examples of Bad
I've filed it along with examples of Bad Statistics because the issue is of chi-squared into a probability is not quite as easy as it may seem. Yes, we all know better now, but I don't think its one or the other. all those weight loss ads are junk you are what you eat.plain and simple,Thanks loving seem adult
[ link to this | view in chronology ]
inc, 4 Jun 2009 @ 10:24pm

if you are collecting bad stats it the fault of the person collecting the stats. I notice a lot of times surveys are forced upon you when you don't want to be filling one out. This is a perfect example of when you would get back answers that are way off base.
[ link to this | view in chronology ]
Anonymous Coward, 4 Jun 2009 @ 11:53pm

News flash: people get annoyed when you hassle them to do surveys they have no interest in. Of course they will answer with a load of BS. Of course if you just make it voluntary, only those with a vested interest will respond, and the results will be biased.
The obvious outcome of all this?...Media should stop concocting and reporting these stats as if they actually mean something.
[ link to this | view in chronology ]
Andrew, 5 Jun 2009 @ 12:41am

Amazing spam
Re:"examples of bad" - that is the most intelligent bit of comment spam I've seen in a long time. Is that autogenerated, do you think?
[ link to this | view in chronology ]
Sam (profile), 5 Jun 2009 @ 1:40am

I should point out that this guy has been writing a column on bad statistics on the BBC News website for months, and is also the author of a rather good book on bad stats called The Tiger that Isn't. He's not citing people responding incorrectly as *the* cause of bad stats, merely one of the many many factors covered in his regular column.
[ link to this | view in chronology ]
Tor (profile), 5 Jun 2009 @ 1:54am

Just use another poll to analyse the first one
A couple of years ago a big Swedish newspaper ran an online poll on their web page with the following question: "Do you answer truthfully to web poll questions?"

Enough said.
[ link to this | view in chronology ]
Anonymous Coward, 5 Jun 2009 @ 3:27am

Junk stats? Junk stats are bad, running with them like they are the word of god is worse.

http://latimesblogs.latimes.com/technology/2009/06/bittorrent-vuze-survey-purchasing-dvd.h tml

I think that is the best example of junk stats possible. Their survey group including almost 65% Vuse users, in a study published by, well, Vuse. Talk about misleading and meaningless junk stats. Yet Carlo, your boss ran with this story like it had come from on high, because it supports his (still unproven) concept that somehow people who download everything are still willing to pay for it, and more often than people who don't.

Really, you should chat with Mike before you put these things up, because it would seem that the two of you just don't agree.
[ link to this | view in chronology ]
Laurent GUERBY, 5 Jun 2009 @ 6:20am

Unemployment
One of the most commonly cited measure in economics is unemployment. But unemployment of 10% does mean that one out of ten person are without job, because of the concept of "active population" which is people with jobs plus people looking "actively" for a job. Each country typically defines 5 to 10 "actively looking" levels and publishes one as "official unemployment". OECD takes all those and publishes a normalized, that is comparable between countries, unemployment level. However a little know fact ...

From OECD normalized numbers for 2007 (latest available):

* men aged 25-54

employment divided by population
France: 88.3% (hence 11.7% jobless)
USA: 87.5% (hence 12.5% jobless)

normalized unemployment rate
France: 6.3%
USA: 3.7%

So with a slightly higher relative working population in this age/sex group France has ... a 70% higher unemployment rate than the USA.

* women aged 25-54

E/P
FR 76.1
USA 72.5

unemployment
FR 7.7
USA 3.8

So here we have clearly more women working in France (3.6 percentage point) relative to their population but unemployment is 102% higher in France!

* conclusion

Employment/population is an objective measure. Unemployment is an extremely subjective measure and so it's easy to get very wide discrepancies as shown above.

* question

Any paper studying this discrepancy in OECD normalized numbers? For male 25-54 there's no real reason not to work, so what are all those USA men doing?

* second question

What credit do you place on "empirical validation" of economic theories (and papers) who use unemployment by country to select the right policy?
[ link to this | view in chronology ]
EEJ (profile), 5 Jun 2009 @ 7:58am

Surveys
Another problem with survey accuracy is the design of the surveys. I've taken several surveys online (in exchange for payment or entries in sweepstakes) and have been very frustrated when questions are asked but don't have enough viable answers available (or a way to explain your answer).

There doesn't seem to be enough middle ground between some of the answers, and sometimes there isn't a viable answer avialable, or the question doesn't apply to me at all, based on previous answers I've given or my current situation.
[ link to this | view in chronology ]
Anonymous Coward, 5 Jun 2009 @ 8:33am

Carlo: You, Yes You, Are To Blame For Junk Posts
[ link to this | view in chronology ]
Ben, 5 Jun 2009 @ 8:33am

My fault
I wonder how much I have affected advertising for our local paper. Because I refuse to divulge my information by creating an account, it frequently (every time I clear my cookies) asks me a few questions before I go to an article. I can just see the ad department trying to figure my habits as a female born in 1901 and living in zip code 90210.
[ link to this | view in chronology ]
porkster, 7 Jun 2009 @ 1:00pm

Guilty as charged!
Yep,
Thats me, giving out bad answers as usual.

If you ring me in the middle of my dinner (actually if you ring me at all and I haven't requested it) of course your going to get a skewed results.

If your web site insists on a pop-up for a survey and it keeps poping up then yes, your going to get cr*p answers.

If you keep sending me e-mail to fill out a survey then you only have yourself to blame.

And one guarenteed lost customer.
[ link to this | view in chronology ]