How To Create A Survey That Makes A Problem Seem Bigger Than It Is: 'Do You Know Anyone...'
from the lying-with-stats dept
There are all sorts of ways to lie with stats and surveys. One of the more obnoxious ones is used in a new "study" about "mobile etiquette" which claims the problem is reaching epidemic levels mainly based on an online survey. The accuracy of online surveys are already suspect enough, but this form of survey is done in a sneaky way that makes the problem look potentially much worse than it is:Nine out of ten people who responded to the online survey said they have seen people misuse their mobile devices, according to the survey.Right. It didn't ask people about their own activities, but just if they've seen other people. That's almost meaningless. Taken to an extreme version, say there was one (highly visible) mobile jackass, and 99 other people surveyed saw that one guy. Well, based on this survey, you could claim that mobile etiquette is in a terrible state.
I'm not arguing that there isn't necessarily a problem with how some people handle mobile etiquette. There could very well be just such a problem, but a study that sets out to investigate the issue with a survey like this one really tells you nothing.
Thank you for reading this Techdirt post. With so many things competing for everyone’s attention these days, we really appreciate you giving us your time. We work hard every day to put quality content out there for our community.
Techdirt is one of the few remaining truly independent media outlets. We do not have a giant corporation behind us, and we rely heavily on our community to support us, in an age when advertisers are increasingly uninterested in sponsoring small, independent sites — especially a site like ours that is unwilling to pull punches in its reporting and analysis.
While other websites have resorted to paywalls, registration requirements, and increasingly annoying/intrusive advertising, we have always kept Techdirt open and available to anyone. But in order to continue doing so, we need your support. We offer a variety of ways for our readers to support us, from direct donations to special subscriptions and cool merchandise — and every little bit helps. Thank you.
–The Techdirt Team
Filed Under: statistics, surveys
Reader Comments
Subscribe: RSS
View by: Time | Thread
Specific case in point: a restaurant I frequent on a weekly basis. Used to grit my teeth at least once every 2 visits in reaction to a loud or annoying ring and/or conversation (and one time 2 young kids watching a YouTube video at full volume). Past 6 months or so? Not a peep.
Standing in line at various stores was always a near guarantee of those one-sided conversations that are too loud - because they're so far away, you see, gotta yell into the phone*. Lately? Can't recall the last time it happened.
I don't see too many folks walking and phoning/texting, but here is not very urban or crowded either.
People who drive like idiots...there are too many reasons to guess at, won't jump to conclusions there.
Maybe increased awareness in my little neck of the woods?
*The spouse was often guilty of this until it was brought to his attention. Now he whispers when he calls from the store with a question or dilemma, lol.
[ link to this | view in chronology ]
I see your point however, and agree the results of the survey are essentially bogus, but I can also see why they may have phrased the question in the way they did. Self-reflection isn't most people's strong suit.
[ link to this | view in chronology ]
[ link to this | view in chronology ]
http://www.techdirt.com/articles/20110228/23515813305/new-study-70-people-find-piracy-soc ially-acceptable.shtml
[ link to this | view in chronology ]
[ link to this | view in chronology ]
Common Practice
[ link to this | view in chronology ]
Re: Common Practice
It's horrible. Absolutely horrible. Even on subjects which are super-obscure, downright impossible if you spend two minutes thinking, or otherwise everyone should be answering 'no', you can get a majority back saying 'yes'.
And as the subject becomes less obscure, the problem becomes worse, obviously. Because less obscure = more visibility.
The white-washing of your data into a nearly normal curve with heavy negative skew should've been the tip that perhaps you are measuring a pseudo-random variable instead of what you want to measure. Unrelated questions on surveys should NOT form any type of pattern reliably.
I don't buy the idea about lying about their own involvement, there must be better ways to ask than that.
[ link to this | view in chronology ]
[ link to this | view in chronology ]
Can't resist the classic Yes Prime MInister.....
Bernard Woolley: "Yes"
Sir Humphrey: "Are you worried about the rise in crime among teenagers?"
Bernard Woolley: "Yes"
Sir Humphrey: "Do you think there is a lack of discipline in our Comprehensive schools?"
Bernard Woolley: "Yes"
Sir Humphrey: "Do you think young people welcome some authority and leadership in their lives?"
Bernard Woolley: "Yes"
Sir Humphrey: "Do you think they respond to a challenge?"
Bernard Woolley: "Yes"
Sir Humphrey: "Would you be in favour of reintroducing National Service?"
Bernard Woolley: "Oh...well, I suppose I might be."
Sir Humphrey: "Yes or no?"
Bernard Woolley: "Yes"
Sir Humphrey: "Of course you would, Bernard. After all you told you can't say no to that. So they don't mention the first five questions and they publish the last one."
Bernard Woolley: "Is that really what they do?"
Sir Humphrey: "Well, not the reputable ones no, but there aren't many of those. So alternatively the young lady can get the opposite result."
Bernard Woolley: "How?"
Sir Humphrey: "Mr. Woolley, are you worried about the danger of war?"
Bernard Woolley: "Yes"
Sir Humphrey: "Are you worried about the growth of armaments?"
Bernard Woolley: "Yes"
Sir Humphrey: "Do you think there is a danger in giving young people guns and teaching them how to kill?"
Bernard Woolley: "Yes"
Sir Humphrey: "Do you think it is wrong to force people to take up arms against their will?"
Bernard Woolley: "Yes"
Sir Humphrey: "Would you oppose the reintroduction of National Service?"
Bernard Woolley: "Yes"
Sir Humphrey: "There you are, you see Bernard. The perfect balanced sample."
[ link to this | view in chronology ]
Daft Surveys
So: 60% of kids are being harmed by their computers;
or, 60% of people believe that kids are being harmed;
or, 60% of people don't know what they are talking about;
or, none of the above.
I think that sounds about right.
[ link to this | view in chronology ]
This just in:
Seems strange they wouldn't attempt some sort of control by asking these respondents if they had abused phone etiquette. My guess is that would fall way under 50%, which would invalidate the first claim.
It's pretty easy to get someone to agree that someone else sucks.
[ link to this | view in chronology ]
Re: This just in:
[ link to this | view in chronology ]
Re: Re: This just in:
Side note: It was much easier to bludgeon people to death with a cell phone back in the early 90s. Smaller components mean less effective bludgeoning, people.
[ link to this | view in chronology ]
Re: Re: Re: This just in:
[ link to this | view in chronology ]
Old hat
[ link to this | view in chronology ]
Overhyped
"Over 1.2 million women and 900,000 men are stalked every year according to The British Crime Survey 2004"
Ok, we're quite a small country, we only have ~66 million people here, so that's really quite a large number!
[ link to this | view in chronology ]
DAMN, and I finally thought mikee wrote something
Then after reading it, I see that little mikee is just bitching and moaning about someone ELSES work, instead of doing something himself.
But then do we REALLY need little mikee making up surveys, hell if he had his way the entire world would be free to him, and noone ever would deserve to earn anything for their work.
[ link to this | view in chronology ]
Re: DAMN, and I finally thought mikee wrote something
[ link to this | view in chronology ]
Re: DAMN, and I finally thought mikee wrote something
[ link to this | view in chronology ]
Re: Re: DAMN, and I finally thought mikee wrote something
I've always kinda thought he was referring to one of his own body parts.
[ link to this | view in chronology ]
Re: Re: DAMN, and I finally thought mikee wrote something
[ link to this | view in chronology ]
Re: DAMN, and I finally thought mikee wrote something
[ link to this | view in chronology ]
No. Quantitative studies in sociology are nothing like this, and sociologists are quite capable of creating a study which is significantly more objective. This "survey" is junk science, and no professional sociologist would ever draw the conclusions about behavior from the indicated results. All one could reasonably do would be to draw conclusions about people's *impressions* of behavior. There is a big difference - obviously.
Unfortunately, roughly 2% of the news media is able to make that differentiation, and as a result, we're constantly bombarded by horribly misleading and inflammatory information. Good for ratings. Bad for everyone.
Kudos for calling BS on this, and for all of the right reasons.
[ link to this | view in chronology ]
Re:
The way the other guy said it, he was a sociologist, or studied as one, and that was a common technique. Now I won't have to track down a soc. prof here and badger him with questions. :p
Yeah, I studied pure math, (and some stats with that), and I'm the guy constantly complaining about survey questions in politics. The religion question on British surveys, for a recent and strong example.
Or, y'know, the conservatives up in Canada cutting the long-form census completely. I suppose it doesn't hurt them any, since they mainly pull their stats from their . . . hats.
[ link to this | view in chronology ]
[ link to this | view in chronology ]
Survey
It is very hard to write a Survey that is completely objectionable (non biased).
Case in point our upper management approved a survey to rate their own performance for the purpose of their own bonus. No matter how you answer the survey question you put upper management in sparkling light (and not of the type of golden showers they rain down on you). For example: Does making your clients happy make you happy?
If you really want to know the real answers you need to find a very good third party to masterfully construct the survey questions. Sadly many survey writers lack the required skills.
Just because you want to know X does not necessarily mean you can directly ask X. The skill is exact wording and associated questions that can tell you exact about X.
A bad survey is like a account saying.... "and what do you want those numbers to add up to?"
[ link to this | view in chronology ]
Re: Survey
Do they really want to know X ?
Or are they simply looking for agreement with their view?
Typically, surveys have an angle and attempt to persuade via herd mentality.
[ link to this | view in chronology ]
I always do everything I can to put the phone down and avoid it when my kids are present, their attention and my attention is vastly more important. I'm constantly asking my kids to tell me something unique about their day at school, and to engage them in conversation about everything and everything that might spark other conversation.
I also see kids with their heads in these devices, and I don't really blame them for long cartrips, but I see them basically stay in cars when their family goes into a store. I guess I have a concern that these devices cull some level of social development that is part of growing up and may be useful later in life.
We limit our kids media to 2 hours a day on non-school days, and more if the media is "educationally based."
[ link to this | view in chronology ]