How To Create A Survey That Makes A Problem Seem Bigger Than It Is: 'Do You Know Anyone...'

from the lying-with-stats dept

Wed, Mar 16th 2011 9:13pm — Mike Masnick

There are all sorts of ways to lie with stats and surveys. One of the more obnoxious ones is used in a new "study" about "mobile etiquette" which claims the problem is reaching epidemic levels mainly based on an online survey. The accuracy of online surveys are already suspect enough, but this form of survey is done in a sneaky way that makes the problem look potentially much worse than it is:

Nine out of ten people who responded to the online survey said they have seen people misuse their mobile devices, according to the survey.

Right. It didn't ask people about their own activities, but just if they've seen other people. That's almost meaningless. Taken to an extreme version, say there was one (highly visible) mobile jackass, and 99 other people surveyed saw that one guy. Well, based on this survey, you could claim that mobile etiquette is in a terrible state.

I'm not arguing that there isn't necessarily a problem with how some people handle mobile etiquette. There could very well be just such a problem, but a study that sets out to investigate the issue with a survey like this one really tells you nothing.

Thank you for reading this Techdirt post. With so many things competing for everyone’s attention these days, we really appreciate you giving us your time. We work hard every day to put quality content out there for our community.

Techdirt is one of the few remaining truly independent media outlets. We do not have a giant corporation behind us, and we rely heavily on our community to support us, in an age when advertisers are increasingly uninterested in sponsoring small, independent sites — especially a site like ours that is unwilling to pull punches in its reporting and analysis.

While other websites have resorted to paywalls, registration requirements, and increasingly annoying/intrusive advertising, we have always kept Techdirt open and available to anyone. But in order to continue doing so, we need your support. We offer a variety of ways for our readers to support us, from direct donations to special subscriptions and cool merchandise — and every little bit helps. Thank you.

–The Techdirt Team

Filed Under: statistics, surveys

28 Comments

If you liked this post, you may also be interested in...

Reader Comments

Subscribe: RSS

View by: Time | Thread

Anonymous Coward, 16 Mar 2011 @ 9:45pm

Maybe I don't get out enough, but I've actually noticed an improvement in mobile device behavior compared to this time maybe a year ago.

Specific case in point: a restaurant I frequent on a weekly basis. Used to grit my teeth at least once every 2 visits in reaction to a loud or annoying ring and/or conversation (and one time 2 young kids watching a YouTube video at full volume). Past 6 months or so? Not a peep.

Standing in line at various stores was always a near guarantee of those one-sided conversations that are too loud - because they're so far away, you see, gotta yell into the phone*. Lately? Can't recall the last time it happened.

I don't see too many folks walking and phoning/texting, but here is not very urban or crowded either.

People who drive like idiots...there are too many reasons to guess at, won't jump to conclusions there.

Maybe increased awareness in my little neck of the woods?

*The spouse was often guilty of this until it was brought to his attention. Now he whispers when he calls from the store with a question or dilemma, lol.
[ link to this | view in thread ]
lux (profile), 16 Mar 2011 @ 9:48pm

Hm, I doubt most people who violate the basics of phone etiquette would implicate themselves in a survey. Am I wrong or do these people seem oblivious to their own behavior? Not sure they're the right folks to be asking.

I see your point however, and agree the results of the survey are essentially bogus, but I can also see why they may have phrased the question in the way they did. Self-reflection isn't most people's strong suit.
[ link to this | view in thread ]
Liz, 16 Mar 2011 @ 9:57pm

I'd like to start a poll asking if anyone knows of a corrupt politician.
[ link to this | view in thread ]
Anonymous Coward, 16 Mar 2011 @ 10:01pm

It isn't just the question, it is often the slanted person reading the study or survey:

http://www.techdirt.com/articles/20110228/23515813305/new-study-70-people-find-piracy-soc ially-acceptable.shtml
[ link to this | view in thread ]
Anonymous Coward, 16 Mar 2011 @ 10:18pm

Josh Smith is the new John Smith?
[ link to this | view in thread ]
Anonymous Coward, 16 Mar 2011 @ 10:33pm

Common Practice
This is actually common practice in sociological survey's conducted about sensitive material where the subject is likely to lie about their own involvement. An example would be "Do you know anyone who would refuse to have a homosexual roommate?" While it seems like it would skew the data (and to some measurable degree it does), it allows you to investigate otherwise unbreachable data sets. This is often done regarding taboo topics such as race / sexuality / crime.
[ link to this | view in thread ]
charles, 16 Mar 2011 @ 10:37pm

I have often thought about a poll question such as: do people speed in front of your house. They usually will answer in the affirmative with a !. This is not very conclusive but you would really see a pattern on a poll.
[ link to this | view in thread ]
freak (profile), 17 Mar 2011 @ 1:07am

Re: Common Practice
Is it the study of sociology that encourages this?

It's horrible. Absolutely horrible. Even on subjects which are super-obscure, downright impossible if you spend two minutes thinking, or otherwise everyone should be answering 'no', you can get a majority back saying 'yes'.

And as the subject becomes less obscure, the problem becomes worse, obviously. Because less obscure = more visibility.

The white-washing of your data into a nearly normal curve with heavy negative skew should've been the tip that perhaps you are measuring a pseudo-random variable instead of what you want to measure. Unrelated questions on surveys should NOT form any type of pattern reliably.

I don't buy the idea about lying about their own involvement, there must be better ways to ask than that.
[ link to this | view in thread ]
Not an Electronic Rodent (profile), 17 Mar 2011 @ 2:14am

Can't resist the classic Yes Prime MInister.....
Sir Humphrey: "You know what happens: nice young lady comes up to you. Obviously you want to create a good impression, you don't want to look a fool, do you? So she starts asking you some questions: Mr. Woolley, are you worried about the number of young people without jobs?"
Bernard Woolley: "Yes"
Sir Humphrey: "Are you worried about the rise in crime among teenagers?"
Bernard Woolley: "Yes"
Sir Humphrey: "Do you think there is a lack of discipline in our Comprehensive schools?"
Bernard Woolley: "Yes"
Sir Humphrey: "Do you think young people welcome some authority and leadership in their lives?"
Bernard Woolley: "Yes"
Sir Humphrey: "Do you think they respond to a challenge?"
Bernard Woolley: "Yes"
Sir Humphrey: "Would you be in favour of reintroducing National Service?"
Bernard Woolley: "Oh...well, I suppose I might be."
Sir Humphrey: "Yes or no?"
Bernard Woolley: "Yes"
Sir Humphrey: "Of course you would, Bernard. After all you told you can't say no to that. So they don't mention the first five questions and they publish the last one."
Bernard Woolley: "Is that really what they do?"
Sir Humphrey: "Well, not the reputable ones no, but there aren't many of those. So alternatively the young lady can get the opposite result."
Bernard Woolley: "How?"
Sir Humphrey: "Mr. Woolley, are you worried about the danger of war?"
Bernard Woolley: "Yes"
Sir Humphrey: "Are you worried about the growth of armaments?"
Bernard Woolley: "Yes"
Sir Humphrey: "Do you think there is a danger in giving young people guns and teaching them how to kill?"
Bernard Woolley: "Yes"
Sir Humphrey: "Do you think it is wrong to force people to take up arms against their will?"
Bernard Woolley: "Yes"
Sir Humphrey: "Would you oppose the reintroduction of National Service?"
Bernard Woolley: "Yes"
Sir Humphrey: "There you are, you see Bernard. The perfect balanced sample."
[ link to this | view in thread ]
Rob Crompton, 17 Mar 2011 @ 2:50am

Daft Surveys
In a discussion a couple of nights ago about use of the internet 3 out of five were pretty definite that youngsters spend all their time in front of their computers playing mindless games. Students lift info from the net and paste it into essay without having a clue what they are on about. And computer tech alters brain structures. 2 out of five disagreed very strongly.
So: 60% of kids are being harmed by their computers;
or, 60% of people believe that kids are being harmed;
or, 60% of people don't know what they are talking about;
or, none of the above.
I think that sounds about right.
[ link to this | view in thread ]
Capitalist Lion Tamer (profile), 17 Mar 2011 @ 2:59am

This just in:
9 out of 10 people believe that other people are worse than they are.

Seems strange they wouldn't attempt some sort of control by asking these respondents if they had abused phone etiquette. My guess is that would fall way under 50%, which would invalidate the first claim.

It's pretty easy to get someone to agree that someone else sucks.
[ link to this | view in thread ]
Not an Electronic Rodent (profile), 17 Mar 2011 @ 3:45am

Re: This just in:
Seems strange they wouldn't attempt some sort of control by asking these respondents if they had abused phone etiquette.
Or indeed some questions to establish exactly what the person they were asking thought "phone etiquette" actually included. I don't recall ever having seen a set of written rules or anything. Ask a few hundred people that and you'd get anything from people who thought getting your phone out in public was bad to people who thought anything short of bludgeoning the guy next to you to death with it was fine.
[ link to this | view in thread ]
Capitalist Lion Tamer (profile), 17 Mar 2011 @ 4:53am

Re: Re: This just in:
Exactly.

Side note: It was much easier to bludgeon people to death with a cell phone back in the early 90s. Smaller components mean less effective bludgeoning, people.
[ link to this | view in thread ]
Fickelbra (profile), 17 Mar 2011 @ 5:04am

Old hat
This is really nothing new, statistics by and large are ways of lying to people via mathematics. I'm paraphrasing but one of my favorite sayings is "Statistics are often used as a drunk man uses a lamp post; for stability rather than illumination". Real non-biased statistics and surveys are rarely seen.
[ link to this | view in thread ]
Daniel Morritt (profile), 17 Mar 2011 @ 5:29am

Overhyped
Stats are just ridiculous, I mean from http://www.nss.org.uk/about/stalking-facts-figures/, anyone that takes these "facts" at face value is a moron.

"Over 1.2 million women and 900,000 men are stalked every year according to The British Crime Survey 2004"

Ok, we're quite a small country, we only have ~66 million people here, so that's really quite a large number!
[ link to this | view in thread ]
Michial Thompson, 17 Mar 2011 @ 5:47am

DAMN, and I finally thought mikee wrote something
Damn, I saw the title and thought "WOW little mikee finally wrote an article on something that he's and expert on."

Then after reading it, I see that little mikee is just bitching and moaning about someone ELSES work, instead of doing something himself.

But then do we REALLY need little mikee making up surveys, hell if he had his way the entire world would be free to him, and noone ever would deserve to earn anything for their work.
[ link to this | view in thread ]
Benjamin (profile), 17 Mar 2011 @ 6:09am

>Is it the study of sociology that encourages this?

No. Quantitative studies in sociology are nothing like this, and sociologists are quite capable of creating a study which is significantly more objective. This "survey" is junk science, and no professional sociologist would ever draw the conclusions about behavior from the indicated results. All one could reasonably do would be to draw conclusions about people's *impressions* of behavior. There is a big difference - obviously.

Unfortunately, roughly 2% of the news media is able to make that differentiation, and as a result, we're constantly bombarded by horribly misleading and inflammatory information. Good for ratings. Bad for everyone.

Kudos for calling BS on this, and for all of the right reasons.
[ link to this | view in thread ]
abc gum, 17 Mar 2011 @ 6:11am

I don't get it. What purpose is this "survey" supposed to serve? Usually, the results are followed by a "therefore we all need to do (whatever)".
[ link to this | view in thread ]
abc gum, 17 Mar 2011 @ 6:12am

Re: DAMN, and I finally thought mikee wrote something
Rather than complaining about the lack of contribution by others, maybe you could concentrate on providing some yourself?
[ link to this | view in thread ]
Jimr (profile), 17 Mar 2011 @ 6:18am

Survey
Survey only tell you what you want to hear.
It is very hard to write a Survey that is completely objectionable (non biased).
Case in point our upper management approved a survey to rate their own performance for the purpose of their own bonus. No matter how you answer the survey question you put upper management in sparkling light (and not of the type of golden showers they rain down on you). For example: Does making your clients happy make you happy?

If you really want to know the real answers you need to find a very good third party to masterfully construct the survey questions. Sadly many survey writers lack the required skills.

Just because you want to know X does not necessarily mean you can directly ask X. The skill is exact wording and associated questions that can tell you exact about X.

A bad survey is like a account saying.... "and what do you want those numbers to add up to?"
[ link to this | view in thread ]
Marcus Carab (profile), 17 Mar 2011 @ 6:28am

Re: DAMN, and I finally thought mikee wrote something
Who is "little mikee"?
[ link to this | view in thread ]
abc gum, 17 Mar 2011 @ 6:42am

Re: Survey
"Just because you want to know X"

Do they really want to know X ?
Or are they simply looking for agreement with their view?

Typically, surveys have an angle and attempt to persuade via herd mentality.
[ link to this | view in thread ]
Gwiz (profile), 17 Mar 2011 @ 7:09am

Re: Re: DAMN, and I finally thought mikee wrote something
Who is "little mikee"?

I've always kinda thought he was referring to one of his own body parts.
[ link to this | view in thread ]
Capitalist Lion Tamer (profile), 17 Mar 2011 @ 7:40am

Re: Re: DAMN, and I finally thought mikee wrote something
I'm more concerned with Mr./Mrs. Noone, who apparently are having trouble getting people to pay for whatever the hell it is that they make.
[ link to this | view in thread ]
freak (profile), 17 Mar 2011 @ 7:51am

Re:
Glad to hear that Sociology doesn't really form conclusions from such data.

The way the other guy said it, he was a sociologist, or studied as one, and that was a common technique. Now I won't have to track down a soc. prof here and badger him with questions. :p

Yeah, I studied pure math, (and some stats with that), and I'm the guy constantly complaining about survey questions in politics. The religion question on British surveys, for a recent and strong example.
Or, y'know, the conservatives up in Canada cutting the long-form census completely. I suppose it doesn't hurt them any, since they mainly pull their stats from their . . . hats.
[ link to this | view in thread ]
Howard the Duck, 17 Mar 2011 @ 9:20am

Re: DAMN, and I finally thought mikee wrote something
Or the entire world would be rich, finally paying attention to consumers.
[ link to this | view in thread ]
Not an Electronic Rodent (profile), 17 Mar 2011 @ 9:24am

Re: Re: Re: This just in:
Smaller components mean less effective bludgeoning, people.
Hmm good point. Some sort of stick attachment to make a mace-type arrangment would seem to be in order then? :-)
[ link to this | view in thread ]
Anonymous Coward, 17 Mar 2011 @ 12:35pm

One thing that bugs me is I take my kids to a dance studio, and I'm typically the only guy there. Most of the moms are sitting there with iPhones and iPads completely ignoring their kids finishing up practice or even just ignoring them as they wait for their class.

I always do everything I can to put the phone down and avoid it when my kids are present, their attention and my attention is vastly more important. I'm constantly asking my kids to tell me something unique about their day at school, and to engage them in conversation about everything and everything that might spark other conversation.

I also see kids with their heads in these devices, and I don't really blame them for long cartrips, but I see them basically stay in cars when their family goes into a store. I guess I have a concern that these devices cull some level of social development that is part of growing up and may be useful later in life.

We limit our kids media to 2 hours a day on non-school days, and more if the media is "educationally based."
[ link to this | view in thread ]