Does Being More Vocal In Video Game Violence Debate Mean You Have The Better Argument?

from the quantity-vs.-quality? dept

Mon, Apr 25th 2011 9:09am — Mike Masnick

A few folks sent over the news of some really bizarre research done by Brad Bushman and Craig Anderson on the question of whether or not violent video games harm teens. First of all, the research is already somewhat suspect, in that Anderson has a long history of claims about how violent video games must harm children based on questionable data. This new report is based on such questionable and loop methodology, you almost wonder why they even bothered.

What they did was take the amici briefs from the Supreme Court case concerning California's anti-violent video game law, and run some numbers on who wrote the briefs and how many "published" studies they had. And that was how they determined which one was more credible. I'm not kidding. Quantity over quality:

The researchers analyzed the credentials of the 115 people who signed the Gruel brief, who believe video violence is harmful, and the 82 signers of the Millett brief, who believe video violence is not harmful. (The briefs are named after the lead attorneys for each side.)

The data for the study came from the PsycINFO database, which provides more than 3 million references to the psychological literature from the 1800s to the present, including peer-reviewed journal articles, book chapters or essays, and books.

For each of the signers of the two briefs, the researchers calculated how many articles and books they published on issues relating to violence and aggression in general and on media violence specifically.

The results showed that 60 percent of the Gruel brief signers (who believe video game violence is harmful) have published at least one scientific study on aggression or violence in general, compared to only 17 percent of the Millett brief signers.

Moreover, when the researchers looked specifically at the subject of media violence, 37 percent of Gruel brief signers have published at least one study in that area, compared to just 13 percent of the Millett brief signers.

And they claim that this is "a very objective approach." It's also a profoundly meaningless approach. In case you didn't follow it, there were a ton of amici briefs filed by various parties in this case. This study picked just two of the briefs. The first one (pdf) filed by California State Senator Leland Yee (whom, I believe, may have written the legislation in question), the California Chapter of the American Academy of Pediatrics and the California Psychological Association. That brief supports California's position in the case. The second one (pdf) is a brief from "social scientists, medical scientists and media effects scholars," which is the one that supports the other side, saying that the law isn't constitutional. You can read the two briefs that I linked to above, and you can judge the relative merits of both.

But that's not what Bushman and Anderson did. They simply took the signers of each brief and measured how many of them have published studies on this specific question. Of course, that's a meaningless and arbitrary number, especially when presented in percentages. Based on this methodology, it would mean that if only one person signed the amicus brief, but had published research, then that one would clearly be the most credible, since 100% of the signers would have published. Obviously, that makes no sense.

Now, Bushman and Anderson -- clearly expecting the quantity over quality issue to make for easy mockery of such a ridiculous study -- also added a second element to try to show "quality" as well:

In a further analysis, Bushman and Anderson examined where the signers of both briefs have published their research. The best academic journals have the highest standards and the most rigorous peer review, so only the best research should be published there, Bushman said.

The researchers used a well-established formula, called the impact factor, to determine the top-tier journals, and then calculated how many signers had published in these journals.

Results showed that signers of the Gruel brief had published over 48 times more studies in top-tier journals than did those who signed the Millett brief.

But, again, this attempt at showing "quality" is really a "quantity" study in disguise. It's not looking at the actual credibility of any of the studies, but trying to create an aggregate (but meaningless) number. And, again, the entire basis of this result is a meaningless dataset. I'm really wondering who would possibly read this and think that the results are credible.

Oh yeah, and one final point. Guess what two academics signed that first brief? You guessed it: Craig Anderson and Brad Bushman. Talk about researcher objectivity huh? They create a bogus methodology to try to "prove" that the brief they signed is more credible than someone else's brief? Honestly, when they present methodology like this, it serves mostly to raise questions about their methodology on any other study as well. They've made it clear that they're not researching the truth. They're starting with an established position and trying to figure out ways to present evidence to support that. That's not science.

Thank you for reading this Techdirt post. With so many things competing for everyone’s attention these days, we really appreciate you giving us your time. We work hard every day to put quality content out there for our community.

Techdirt is one of the few remaining truly independent media outlets. We do not have a giant corporation behind us, and we rely heavily on our community to support us, in an age when advertisers are increasingly uninterested in sponsoring small, independent sites — especially a site like ours that is unwilling to pull punches in its reporting and analysis.

While other websites have resorted to paywalls, registration requirements, and increasingly annoying/intrusive advertising, we have always kept Techdirt open and available to anyone. But in order to continue doing so, we need your support. We offer a variety of ways for our readers to support us, from direct donations to special subscriptions and cool merchandise — and every little bit helps. Thank you.

–The Techdirt Team

Filed Under: brad bushman, craig anderson, methodology, studies, video games, violence

35 Comments

If you liked this post, you may also be interested in...

Reader Comments

Subscribe: RSS

View by: Time | Thread

Anonymous Coward, 25 Apr 2011 @ 9:22am

I picture them wearing lab coats with clip boards rigorously counting up published authors and nodding to themselves gravely when they find one who is published and shaking their heads knowingly when they find one who is not published.
[ link to this | view in chronology ]
John Doe, 25 Apr 2011 @ 9:24am

Unfortunately that is what passes for science these days...
"They're starting with an established position and trying to figure out ways to present evidence to support that. That's not science."

This is what passes for science in many circles today, not the least of which is the whole global warming lie.
[ link to this | view in chronology ]
- Anonymous Coward, 25 Apr 2011 @ 10:03am
  
  Re: Unfortunately that is what passes for science these days...
  I'm sure the inferior primates who don't grasp global warming believe so. Happily, they are being out-evolved by superior human beings.
  [ link to this | view in chronology ]
  - John Doe, 25 Apr 2011 @ 11:26am
    
    Re: Re: Unfortunately that is what passes for science these days...
    Primates don't grasp much of anything except bananas. You do realize that CO2 according to the GW freaks, only accounts for a fraction of a percentage of actual warming? So we are spending billions to tackle what, 1/2%? Solar activity and atmospheric water vapor accounts for the overwhelming majoring of warming and we have no control over solar activity and near zero understanding of why the atmosphere isn't holding more water vapor than it currently does.
    
    Besides, I am sure you superior human beings will drag us inferior primates along with your superior grasp of the ways of the world. So we inferior primates may be more evolved than the superior human beings since we can sit back and let you do all the work an yet we reap the same benefits.
    [ link to this | view in chronology ]
    - The Infamous Joe (profile), 25 Apr 2011 @ 11:42am
      
      Re: Re: Re: Unfortunately that is what passes for science these days...
      Got a link handy that has more info on that global warming data? I'm interested in getting the whole picture. If not, I'm sure google will be my friend.
      [ link to this | view in chronology ]
- Anonymous Coward, 25 Apr 2011 @ 12:28pm
  
  Re: Unfortunately that is what passes for science these days...
  Way to hijack the thread jackass...
  [ link to this | view in chronology ]
  - John Doe, 25 Apr 2011 @ 1:50pm
    
    Re: Re: Unfortunately that is what passes for science these days...
    Not really a hijack, just a controversial case in point. But nice way to contribute yourself there. Try to keep the name calling down in the future and you will do better in debates.
    [ link to this | view in chronology ]
Anonymous Coward, 25 Apr 2011 @ 9:30am

[...]They're not researching the truth. They're starting with an established position and trying to figure out ways to present evidence to support that. That's not science.

Sounds like typical modern "science" to me. It's a shame there aren't more people genuinely trying to understand the world, instead of people like these hired by some cause or another to fabricate proof.
[ link to this | view in chronology ]
chuck, 25 Apr 2011 @ 9:31am

This whole issue makes me so angry, I want to just log in and PVP all afternoon.
[ link to this | view in chronology ]
- Josh in CharlotteNC (profile), 25 Apr 2011 @ 9:38am
  
  Re:
  Go work that aggression out in a non-harmful way.
  
  Without video games you might have to go shoot some lawyers and academics in real life.
  [ link to this | view in chronology ]
Atkray (profile), 25 Apr 2011 @ 9:59am

The worst thing is they are collecting a paycheck for this nonsense.
[ link to this | view in chronology ]
The Infamous Joe (profile), 25 Apr 2011 @ 10:00am

Apples and oranges
Am I to understand that one side says violent games are harmful to kids and the other side says a law banning them would be unconstitutional? Why would you need to publish a paper on violence to know about the first amendment? They aren't even comparing the same things. Tomfoolery, I say!
[ link to this | view in chronology ]
- Christopher (profile), 25 Apr 2011 @ 10:28am
  
  Re: Apples and oranges
  More like apples and watermelons! They aren't even the same size.
  
  For goodness sakes, I have played numerous exceedingly bloody games, as have my children and relatives children from younger than when I started.... we are goddamned near PACIFISTS!
  
  That's right..... we turn the other cheek so damned often (except when using words to snipe, we are masters at that) that we have been PHYSICALLY INJURED because we didn't want to fight back when physical force was used against us in real life.
  [ link to this | view in chronology ]
  - I-blz, 25 Apr 2011 @ 1:27pm
    
    Re: Re: Apples and oranges
    No sarcasm, I commend you. I wish I was so peaceful. Hellova lot easi to get through a day that's for sure.
    [ link to this | view in chronology ]
MrWilson, 25 Apr 2011 @ 10:04am

Another issue with the quantity approach is that they didn't bother to read the actual published studies which they counted. Those 60% who supported the Gruel brief who published could have written, "anger bad, games make anger worse, grr," for all Bushman and Anderson know.
[ link to this | view in chronology ]
MVGuy, 25 Apr 2011 @ 10:30am

More hack science
Indeed, that the lead authors of this paper were the lead authors of one of the briefs ought to raise a red flag this is nothing close to "objective"
[ link to this | view in chronology ]
Jesse (profile), 25 Apr 2011 @ 10:32am

This is an obvious outcome of the "publish or perish" mentality of the scientific community. Unfortunately, the focus in academia is in fact on quantity over quality.
[ link to this | view in chronology ]
- Jesse (profile), 25 Apr 2011 @ 10:34am
  
  Re:
  [I submitted prematurely]
  
  Naturally, many researchers assume that those who write more papers on a topic are more credible, rather than those who publish higher quality research less frequently.
  [ link to this | view in chronology ]
Mr. Smarta**, 25 Apr 2011 @ 11:00am

Nothing new... again
Arguments like this spawned back in the 70s and 80s when people were absolutely sure that playing Dungeons and Dragons led to worshipping the devil, performing satanic rituals involving human sacrifice, and butchering and murdering people.

Does that mean all the people who did an all-night D&D hack fest are the very individuals sitting on death row??? Wow! What a concept!! I think I've just figured this all out. Amazing...
[ link to this | view in chronology ]
- The eejit (profile), 25 Apr 2011 @ 11:16am
  
  Re: Nothing new... again
  Hey, I'm a Discordian, but that only came after I started playing D&D. Does that make me an epic level Cleric?
  [ link to this | view in chronology ]
- The Groove Tiger (profile), 25 Apr 2011 @ 11:38am
  
  Re: Nothing new... again
  Not just that, they actually believed that the D&D books were actual instructions on how to cast black magic. But no matter how many times I multiply my level by 1d6 I can't manage to cast a fireball...
  [ link to this | view in chronology ]
  - Michael Price (profile), 29 Jun 2011 @ 6:02am
    
    Re: Re: Nothing new... again
    The weird thing is if they really believed we could do the stuff our characters could do why would they want to fuck witnh us? I mean seriously considering what my characters can do why would a suburban housewife not just go, "Ok do what you want just don't summon a basilisk in my living room."? Of course that wouldn't be necessary, they would just realise we're not the hobbyists they're looking for.
    [ link to this | view in chronology ]
trrll (profile), 25 Apr 2011 @ 11:09am

So one's propensity to support the proposition that video game induced violence is a serious problem is correlated with one's number of publications in the area--which probably also correlates with the amount of one's funding predicated on the proposition that violence in video games is a serious problem justifying continued investment in such research. That doesn't prove that they are wrong, but it sure doesn't prove that they are right.

The real problem, of course, is that video game sales have increased, particularly to the young male demographic that is statistically most associated with violent behavior, and as video games have become more realistically violent, the incidence of violent crime has steadily decreased. That does not, of course, disprove the notion that there is a "pro-violence" influence of video games. What it does prove is that any such effect (if it exists at all) must be so small as to be swamped by other social and demographic factors influencing violent behavior.
[ link to this | view in chronology ]
Jason, 25 Apr 2011 @ 11:37am

Answers in Genesis Anyone?
This sounds like the intelligent design folks who decided there was a creator and then went and found enough evidence presented the right way to make their point (without ever proving their original assumption). It's a near perfect system for folks who want to believe this already and aren't willing to actually ask a critical question or even question what the authority has told them to beleive.

See the trick is to present the right "evidence" ingore anything to the contrary and use big science like words to sound like you are smart. Like lambs to theslaughter.
[ link to this | view in chronology ]
Anonymous Coward, 25 Apr 2011 @ 12:33pm

Besides such wretched practices going on, and I don't know if this stuff is BECAUSE it's a high profile, or if it is just so common that we hear about it because it's high profile. I can't wait to see what the newest violence generator is (I really hope it goes to musical tv shows/movies). As more people who were kids that grew up on these violent video games turn into nice productive adults, this little myth will wither away. If I followed the statistics (the most abused area of math, poor thing) from fear mongering crap like this, I should be a mass murderer by now. Let's see, violent video games? Check, my father bought me video games because they were on the news for being so violent because they looked cool (carmegeddon anyone?). He knew me and knew I understood the difference between a game and reality, I'm saddened to doubt how many parents could honestly say that. Anyways, Anti-social? Yeppers, it got me two degrees electrical engineering and computer science. Oh, and I've done several martial arts for over 17 years, that has to be more violent than the pretty little lights right? I should be foaming at the mouth by now with rage like a bad zombie movie.
[ link to this | view in chronology ]
- Anonymous Coward, 26 Apr 2011 @ 4:57am
  
  Response to: Anonymous Coward on Apr 25th, 2011 @ 12:33pm
  Aggravated assault and battery offences have are at a much higher rate than pre tv/video games. Most people aren't affected by it, but a large enough percentage is that we live in a much more violent society. I worked as a paramedic and can tell you first hand that the rate of murder would be astronomical if it weren't for modern medicine saving so many people. You did martial arts which specifically teaches you discipline. Go on YouTube and have a look at all the morons out there happily slapping each other around - you'll notice they aren't martial arts practitioners and they have no discipline. I call them 'urban ninjas' - a sarcastic name of course.
  [ link to this | view in chronology ]
  - sam, 26 Apr 2011 @ 6:20am
    
    Re: Response to: Anonymous Coward on Apr 25th, 2011 @ 12:33pm
    Do you have any evidence or did you just make all that up? And no anecdotal "I'm a paramedic" isn't evidence, and neither is "oh look some youtube videos".
    
    http://bjs.ojp.usdoj.gov/content/glance/cv2.cfm - violent crime has clearly been trending down since the release of the original Playstation.
    
    And you really think that TV/video games are significant driver of violent crime rates, as opposed to say the war on drugs?
    [ link to this | view in chronology ]
    - Anonymous Coward, 26 Apr 2011 @ 9:17pm
      
      Re: Re: Response to: Anonymous Coward on Apr 25th, 2011 @ 12:33pm
      Sam, I do have evidence - but lets first look at yours. Your report is a combination of two reports - one which is actual crime as reported by official police statistics - the other is a set of interviews with people who give anecdotal evidence so they can make an estimate for probable victimisation of crime. I'm don't mind estimates - just not ones based on anecdotes.
      
      Looking at the two separate graphs - actual crime has steadily increased but estimated crime has dramatically decreased. Combining the results is just total bullshit - a combination of an anecdotal estimate and actual figures is meaningless. Anyway - the graph clearly shows actual real life crime has gone up. It helps if you read the data first ;)
      
      Now - my anecdote was about the ability of medicine to decrease the murder rate by saving people that 40 years ago would have died. Now this isn't just an anecdote - it is a fact. Medicine is much more advanced than it was 40 years ago. Quick example - back then a subdural bleed from a punch in the head meant your death - now they'll save you. See http://hsx.sagepub.com/content/6/2/128.abstract for a comprehensive study supporting this (3 out of 4 murders now prevented by medical intervention).
      
      Onto your request for my "evidence". Firstly I'm not obliged to do this - this is a comment section of an article. These are all from Dave Grossman's "On Combat"
      
      Look at: FBI Uniform Crime Report - 1957 to 2000 shows a 5x increase in per capita crime rates.
      
      Consider Australia - http://www.abs.gov.au/ausstats/abs@.nsf/Lookup/4524A092E30E4486CA2569DE00256331
      
      The murder rate from the beginning of the 20th Century until about 1950 halved. People killing less, maybe, more likely medicine saved them. Of course the trend should be continually downward but it isn't. It doubles again to in the next 50 years. Now combine this your new knowledge about medical advances and you can see that it would be 4 times this number without medical advances - or an 8 times increase in murder rate. The only thing stopping this is more advanced medicine.
      
      Now my request. Please show how medicine is less advanced and saves less people than it did 50 years ago. If you can show that I'll eat my words.
      [ link to this | view in chronology ]
      - Yet another anonymous coward, 27 Apr 2011 @ 12:56am
        
        Re: Re: Re: Response to: Anonymous Coward on Apr 25th, 2011 @ 12:33pm
        But if you consider Australia, immigration has significantly changed the population demographic since the 50s. You could also argue that immigration has caused an increase in the murder rate. Which is just as specious an argument as the virtual violence begets real violence camp, because it uses one number to justify another based on no scientific process.
        [ link to this | view in chronology ]
        
        Anonymous Coward, 27 Apr 2011 @ 4:46am
        
        Re: Re: Re: Re: Response to: Anonymous Coward on Apr 25th, 2011 @ 12:33pm
        Actually I happen to live in Australia (hence the Australian data). You're actually on to something. As a paramedic I saw a significantly higher percentage of immigrants with mental illness versus the indigenous (as in anyone born in Aus) population. They had often come from war torn countries and brought with them extremely violent practices in terms of fighting. No more step outside and have a quick punch up, now it's step outside and often - get knifed. Either way - murder rate has gone up, not down, and has been capped at a lower relative rate than it should be through advances in medicine.
        
        Seriously though, suggesting that virtual violence begets real violence is not a specious argument. Exposure to anything at a high enough level both makes you intimate with it and numbs you to it.
        
        Case in point - police, paramedics, firefighters, doctors and nurses all have a high amount of trauma and death around them. The regular population (or people new to the job) don't cope particularly well with it. But with enough exposure you'll be eating your lunch next to a stretcher dripping with blood from the last massive trauma patient and cracking jokes about deceased people - while they're still warm (totally random examples of course...)
        
        So, expose yourself to violence and expect that you'll get pretty intimate with it and numb to the reality of it (until you get the shit kicked out of you because you can't fight - like most urban ninjas). Keep in mind this is a totally accepted general principal - so why oh why are people trying to suspend it with this particular topic?
        
        I'm happy to admit I've been an avid FPS gamer since 1990. I've played just about every good FPS game since then, although it's getting hard to keep up with the amount that are released these days so I'm much more picky choosy now. Do I think it's affected me? You betcha. Twenty years of killing mutant zombies has given me an itchy trigger finger should I ever see one. Same as most people really.
        
        We'll see if people find photorealistic virtual rape games acceptable when they hit the internet. I'm sure the users of it will be adamant that their virtual sickness won't extend into real life. What if it's the rape and/or killing of a virtual likeness of your own husband/wife/kids some freak is beating off to? I'm sure you won't mind since it's virtual (you as in anyone reading this - not the previous poster specifically). Don't worry, 99.9% of the time their obsession won't extend into real life. Good odds hey? Don't worry, this is a specious argument and all ;)
        [ link to this | view in chronology ]
The eejit (profile), 25 Apr 2011 @ 2:28pm

I'm somewhat amused by the BEING LOUDER MEANS THAT YOU ARE RIGHT, EVEN WHEN YOU'RE WRONG YOU MORONIC HEAD ON A STICK.,/B>
[ link to this | view in chronology ]
Anonymous Coward, 26 Apr 2011 @ 7:39am

Not all opinions are equal. Not all science is equal.
I think the Bushman/Anderson analysis is less goofy than it is being spun here.

The problem is that many people, including parents, politicians, and tech blog readers, have opinions on the ways in which media does (or does not) influence attitudes and aggressive behavior. But, because a parent or lawmaker has an opinion, doesn't mean that he or she has a good understanding of the research literature on the topic, a literature that has both strengths and weaknesses. Some people have opinions that are supported by evidence. Some people have opinions that are supported by principle, coupled unrepresentative anecdotes.

What the Bushman/Anderson analysis demonstrates is that the people supporting one particular conclusion on this issue have more credibility: they've done the research themselves in many cases. As such, they are better positioned than you or I to understand how it informs the basic issues.

Sure, if could very well be the case that these authors reached a conclusion long ago and selectively publish studies that support their case. But that is merely an assumption on Mike's part based on a distrust of the "science industry." I don't see any support for the assumption.
[ link to this | view in chronology ]
Beth W., 26 Apr 2011 @ 9:17am

Excellent!
Thanks so much for writing this. It's a subject I tackled recently on my own blog: http://backwardmessages.wordpress.com/2011/04/21/teens-arent-buying-violent-games-but-people-keep-bu ying-bad-science/. It's absolutely ridiculous that these guys tried pass this "study" off as "objective" or even convincing. Anderson is a pro at doing these studies-of-studies thing, too. I wish more folks in the press would dig deeper into these kinds of studies as you have, so we can show the public they have cause for suspicion.
[ link to this | view in chronology ]
Gene Cavanaugh, 26 Apr 2011 @ 12:45pm

Does knowing more about a subject qualify you to give an opinion?
You blew it big time on this one, Michael!

When I first started in engineering, there was an enormous debate about education versus the so-called "school of hard knocks", with the implication that experience was superior to an education. To a lesser extent, I saw this in marketing and other fields as well.

Now we are saying people who have studied the problem carefully (as shown by the papers they have contributed) know less than "knee-jerk" reactionists! That is so naive!!!!
[ link to this | view in chronology ]
fred, 29 Jun 2011 @ 5:42am

McDonalds makes the most hamburgers therefore they are the best.
McDonalds has the most commercials therefore they are the most credible.
I fear for the future with the lack of critical thinking displayed by the general public and many scientists.
[ link to this | view in chronology ]