Sting Operation Shows How Full Of Crap Health Journals Are When It Comes To Dietary Studies
from the eat-it dept
Look, I probably don't have to tell you Techdirt readers this, but I'm a strange sort of cat. I could go into all the reasons why I'm odd, but whenever I try to explain to people how non-normal I am, I usually just reveal this little bit of truth: I hate chocolate. No, I don't not-love chocolate. Nor do I dislike chocolate. I fucking hate it, nearly as much as I hate how low I appeared on this ingenious bit of sleuthing a commenter did in determining which Techdirt writers swear the most (a list which I insist is fucking bullshit, by the way). That said, everyone else loves chocolate, of course, so I'm sure they and many others were thrilled to see so many well-respected publications blaring headlines recently about how chocolate can help reduce weight. I'd show you a bunch of links to those stories put forth by supposedly well-respected journalism outlets and scientific journals that make heavy claims about peer-reviews and fact-checking, but I can't because most of those stories have been pulled. Why?
Because the whole thing was a bullshit hoax put on by a journalist to make the point that, at least when it comes to studies around diet and health, the journals and the media the reports on their papers are largely full of crap. Go read that entire thing, because it's absolutely fascinating, but I'll happily give you the truncated version. John Bohannon, who has a Ph.D in molecular biology of bacteria and is also a journalist, conspired with a German reporter, Peter Onneken, to see how badly they could fool the media to create BS headlines. They did this by turning John Bohannon into Johannes Bohannon (obviously) and creating a website for The Institute of Diet and Health, which isn't actually a thing. Then they conducted a very real study with three groups: 1 group eating a low-carb diet, 1 group eating their regular diet, and 1 group eating a low-carb diet and a 1.5oz bar of dark chocolate daily. After running background on the groups, conducting blood tests to correct for disease and eating disorders, and hiring a German doctor and statistician to perform the study, away they went. The results?
Onneken then turned to his friend Alex Droste-Haars, a financial analyst, to crunch the numbers. One beer-fueled weekend later and... jackpot! Both of the treatment groups lost about 5 pounds over the course of the study, while the control group’s average body weight fluctuated up and down around zero. But the people on the low-carb diet plus chocolate? They lost weight 10 percent faster. Not only was that difference statistically significant, but the chocolate group had better cholesterol readings and higher scores on the well-being survey.Bam, results! Not just results, but results the media would absolutely love to sink their idiotic teeth into. The problem? Well, the method for running the entire study was bullshit.
Here’s a dirty little science secret: If you measure a large number of things about a small number of people, you are almost guaranteed to get a “statistically significant” result. Our study included 18 different measurements—weight, cholesterol, sodium, blood protein levels, sleep quality, well-being, etc.—from 15 people. (One subject was dropped.) That study design is a recipe for false positives.Bohannon goes into some of the gory math, and it really is fun to read, but this is pretty easy to understand. With a small enough sample size and testing for as wide a range of results and factors as possible, you absolutely expect to find greater variance than if your study was testing for less factors or had a higher sample size. It's simple: people are different and testing less people makes those difference statistically appear to be more significant.
Think of the measurements as lottery tickets. Each one has a small chance of paying off in the form of a “significant” result that we can spin a story around and sell to the media. The more tickets you buy, the more likely you are to win. We didn’t know exactly what would pan out—the headline could have been that chocolate improves sleep or lowers blood pressure—but we knew our chances of getting at least one “statistically significant” result were pretty good.
Anyway, the team then went to the International Archives of Medicine, which Bohannon identifies as a "fake journal" publisher. In other words, pay enough Euros and your "study" gets "published", all without the bothersome time-waster known as being peer-reviewed. Not that IAM doesn't claim to be reviewed. It certainly does make that claim, but after payment was accepted Bohannon found that their study had been accepted without change. And, keep in mind, this study is designed to be bad. So, once the study had been published, it was time for the PR machine to swing into action.
Take a look at the press release I cooked up. It has everything. In reporter lingo: a sexy lede, a clear nut graf, some punchy quotes, and a kicker. And there’s no need to even read the scientific paper because the key details are already boiled down. I took special care to keep it accurate. Rather than tricking journalists, the goal was to lure them with a completely typical press release about a research paper. (Of course, what’s missing is the number of subjects and the minuscule weight differences between the groups.) But a good press release isn’t enough. Reporters are also hungry for “art,” something pretty to show their readers. So Onneken and Löbl shot some promotional video clips and commissioned freelance artists to write an acoustic ballad and even a rap about chocolate and weight loss. (It turns out you can hire people on the internet to do nearly anything.)And it didn't take the reporters long to pick up this crap-on-a-stick and run with it like children after the ice cream truck. Not all of them, but some of the stories are still up. The Daily Star covered their paper, for instance, as did the Times of India, international editions of The Huffington Post, and some television news programs. Men's Health was going to go with a story in September, though that probably won't run now. Shape Magazine didn't get off so lucky, with their story appearing in the June issue, in print. And remember, this is all bullshit. None of it is real. How does something like this happen?
Onneken wrote a German press release and reached out directly to German media outlets. The promise of an “exclusive” story is very tempting, even if it’s fake. Then he blasted the German press release out on wire service based in Austria, and the English one went out on NewsWire. There was no quality control. That was left to the reporters.
The answer is lazy "journalists."
When reporters contacted me at all, they asked perfunctory questions. “Why do you think chocolate accelerates weight loss? Do you have any advice for our readers?” Almost no one asked how many subjects we tested, and no one reported that number. Not a single reporter seems to have contacted an outside researcher. None are quoted. These publications, though many command large audiences, are not exactly paragons of journalistic virtue. So it’s not surprising that they would simply grab a bit of digital chum for the headline, harvest the pageviews, and move on. But even the supposedly rigorous outlets that picked the study up failed to spot the holes.Now, there is some humor in all of this, but also danger. It's one thing to claim that chocolate leads to weight loss and have the media run wild with it, but we all know that fad diets and exciting health claims rain down on us in buckets, and I think it's safe to say that not all of them are as harmless as Bohannon's. The average person hasn't done much thinking about the validity of these studies that they read about in the media; they simply trust the media to do the fact-checking. The media, it appears, largely trusts the journals to do the reviews and fact-checking. Except some (many?) of those journals don't. The whole thing harkens back to one of the funnier moments in the Anchorman movie, when the main character makes a ludicrous claim about women's brains being smaller than men's, and then punctuates the statement with a smirk, saying, "It's science." As far as much of the media reporting goes, it might as well be "science."
Strangely, Bohannon notes that readers of the articles were apparently more skeptical than the authors.
There was one glint of hope in this tragicomedy. While the reporters just regurgitated our “findings,” many readers were thoughtful and skeptical. In the online comments, they posed questions that the reporters should have asked.If we've reached a time when readers are more skeptical than the reporters, that's a massive problem for journalism, but perhaps a delightful sign for the spread of skepticism and inquiry amongst the public. Either way, look with a critical eye the next time you hear about that fad diet or health food claim.
“Why are calories not counted on any of the individuals?” asked a reader on a bodybuilding forum. “The domain [for the Institute of Diet and Health web site] was registered at the beginning of March, and dozens of blogs and news magazines (see Google) spread this study without knowing what or who stands behind it,” said a reader beneath the story in Focus, one of Germany’s leading online magazines. Or as one prescient reader of the 4 April story in the Daily Express put it, “Every day is April Fool’s in nutrition.”
Thank you for reading this Techdirt post. With so many things competing for everyone’s attention these days, we really appreciate you giving us your time. We work hard every day to put quality content out there for our community.
Techdirt is one of the few remaining truly independent media outlets. We do not have a giant corporation behind us, and we rely heavily on our community to support us, in an age when advertisers are increasingly uninterested in sponsoring small, independent sites — especially a site like ours that is unwilling to pull punches in its reporting and analysis.
While other websites have resorted to paywalls, registration requirements, and increasingly annoying/intrusive advertising, we have always kept Techdirt open and available to anyone. But in order to continue doing so, we need your support. We offer a variety of ways for our readers to support us, from direct donations to special subscriptions and cool merchandise — and every little bit helps. Thank you.
–The Techdirt Team
Filed Under: chocolate, fooling journalists, hoax, journalism, science, statistics, studies, weight loss
Reader Comments
Subscribe: RSS
View by: Time | Thread
[ link to this | view in chronology ]
Strange Odd and Non-normal
But seriously, I need to talk to you about keeping hollandaise in the refrigerator when it is so easy to make fresh. Just [insert favorite search engine here] for blender hollandaise, it literally takes minutes and has to be better than anything that comes with whatever is necessary to make it stable. Use unsalted butter.
Oh, and having read the headlines for stories about the 'chocolate' diet (I could not bring myself to reading the stories) I thought about all the sugar added to most chocolate and cried 'bullshit' to myself.
[ link to this | view in chronology ]
Re: Strange Odd and Non-normal
[ link to this | view in chronology ]
Unfortunaltely for you, "Life is like a box of chocolates".
[ link to this | view in chronology ]
News Rigging is here...Bot News..
http://ducknetweb.blogspot.com/
[ link to this | view in chronology ]
Re: News Rigging is here...Bot News..
It's strange to complain about news written by bots when the problem in this case is news written by people. Or are you claiming these stories about the chocolate study were not in fact written by people?
[ link to this | view in chronology ]
Re: News Rigging is here...Bot News..
[ link to this | view in chronology ]
[ link to this | view in chronology ]
Re:
I just about choked on my Hershey bar dipped in Lipton chicken noodle cup-a-soup when I read it.
LOL
[ link to this | view in chronology ]
Re:
I like to eat these dry;
http://forthemommas.com/wp-content/uploads/2015/05/Hungry-Jack-Hashbrown-Potatoes.jpg
Or after I've added the water, but before they're cooked.
[ link to this | view in chronology ]
Re: I like eating Lipton chicken noodle cup-a-soup dry, so there Mr. I hate chocolate
[ link to this | view in chronology ]
That said, Timothy please, for the sake of readability look up and embrace the word 'fewer'. Please?
"if your study was testing for less factors or had a higher sample size. It's simple: people are different and testing less people"
[ link to this | view in chronology ]
Re: "less" versus "fewer"
[ link to this | view in chronology ]
Re: Re: "less" versus "fewer"
/smirk
[ link to this | view in chronology ]
Re:
[ link to this | view in chronology ]
Re: Re:
[ link to this | view in chronology ]
Re:
[ link to this | view in chronology ]
[ link to this | view in chronology ]
Re:
His study is dime a dusin and it follows the norms for how to do these experiments, besides the conscious design flaws. Cudos to Bohannon for getting it as right as he did besides the conscious errors. It puts him above most in the field.
As for his conscious errors:
Measuring 18 parameters is crazy. Afaik the rule of thumb is 5 or 6 where you start entering the area of fitting an elephant.
15 testpersons is crazy low in any test. Usually you estimate the natural variance of your study and take in a few extra people for safety.
Calories and physical activity are very often the determining factors in weight loss. Not counting calories is the biggest tell that something is fishy in the study.
[ link to this | view in chronology ]
Re: Re:
First of all, the mythical calorie is a gross oversimplification of complex processes and is an erroneous unit of measure given the fact that everyone's bodies process different types of foods in vastly different ways.
In other words, what might go straight through me might go straight to your gut and may even kill another if they're allergic...
[ link to this | view in chronology ]
Re: Re: Re:
That's why you test with lots of people and look at specifics like standard deviation rather than just one number.
[ link to this | view in chronology ]
[ link to this | view in chronology ]
Not just diet and health
http://pipeline.corante.com/archives/2015/05/12/a_glance_across_the_literature.php
To give you an idea, here's titles for some of the papers:
JACS: "Science Rejected It, and Angewandte Couldn't Think Up a Bad Enough Joke, So Here We Are"
Ang. Chem.: "A Metal-Organic Framework With Nanostructured BODIPY Ligands, Published Without Review on the Basis of the Title Alone"
J. Med. Chem.: "This Project Looks Good, But It Did Not Work. And 18 Out of the 23 Authors have Typographical Symbols Behind Their Names, Because The Work Took Place During Bush's First Term"
[ link to this | view in chronology ]
The actually scary thing is:
So they did not even sell wrong results but rather wrong conclusions.
Their point was not that the science establishment has weak defenses against frauds and charlatans. It was that it is has weak defenses against fools.
[ link to this | view in chronology ]
Re: The actually scary thing is:
[ link to this | view in chronology ]
HEY!
[ link to this | view in chronology ]
S o, a bunch of lazy journalists' laziness came back to bite them in the ass. Their response? "Wah! Those dirty scientists tricked us! They can't experiment on humans!" Nice.
[ link to this | view in chronology ]
Re:
[ link to this | view in chronology ]
Re: Re:
[Side note: Do I want to know what a bronie (brony?) is?]
[ link to this | view in chronology ]
Re: Re: Re:
Also other Ars authors. The very first comment taking issue with some points in the article is an Ars author.
[ link to this | view in chronology ]
A book on this subject
[ link to this | view in chronology ]
I personall have been on an all chocolate
People used to kick sand in my face at the beach, making me unpopular and ridiculed, now they can't kick it high enough to reach my face when I am laying down taking in the sun, my social life has exploded!.
[ link to this | view in chronology ]
Re: I personall have been on an all chocolate
[ link to this | view in chronology ]
Careful
This chocolate study may have been about how easily duped we are, but it's getting wide distribution because it makes people doubt the alternative health information out there. I saw this article in real time and dismissed it because it was so obviously indebted to wishful thinking.
Voluminous amounts of new information about diet are coming out. Much better documented and not fad dieting oriented. Simply put, we've been propagandized into ruining our diets. Why? Because you can make more money from processed foods. But hey, there was a phony story about chocolate, so you'd be stupid to read any of that stuff, which is why you're going to continue to read about this study for a long, long time.
[ link to this | view in chronology ]
People think I'm strange because I hate Summer.
Sure, the trees are all green and look nice, and the sunny sky looks cheerful, but that sunny sky also brings temperatures of 80-90 degrees. On a "cooler" day, a fan might be enough to keep you comfortable while inside, but the second you step outside, you start to sweat. Your hair is wet, your clothes are wet, you smell no matter much deodorant you've doused yourself with. On really hot days, a fan just pushes around hot air and you need an air conditioner to keep from leaving sweat stains on everything you touch. Of course running the air conditioner pretty much doubles your electric bill and walking outside is like stepping into a blast furnace.
Am I the only one who doesn't enjoy sweating, or am I just the only person on the planet who sweats at temperatures above 74 degrees?
[ link to this | view in chronology ]
Re:
[ link to this | view in chronology ]
Re:
I think it's more that a lot of people really really hate cold weather, so they would rather put up with heat than cold. But that's just a guess.
[ link to this | view in chronology ]
Re: Re:
[ link to this | view in chronology ]
[ link to this | view in chronology ]
One misunderstanding here...
[ link to this | view in chronology ]
Re: One misunderstanding here...
[ link to this | view in chronology ]
Chocolate
[ link to this | view in chronology ]
Not just for-pay health journals
http://archpsyc.jamanetwork.com/article.aspx?articleid=1151061
[ link to this | view in chronology ]
Is it really the journalists fault?
While this is a blatant example, most examples of this phenomena will not include obvious problems that a journalist could have spotted.
[ link to this | view in chronology ]
Re: Is it really the journalists fault?
[ link to this | view in chronology ]
Then Again ...
[ link to this | view in chronology ]
RIP
[ link to this | view in chronology ]
less factors... less people... ?
FEWER factors
FEWER people
[ link to this | view in chronology ]
Chocolate hatred
[ link to this | view in chronology ]
Or any social sciences 'study".
[ link to this | view in chronology ]
What is ludicrous about that claim? The science is pretty well established that female brains are on average about 10 % smaller.
[ link to this | view in chronology ]
Re:
[ link to this | view in chronology ]
ethics
[ link to this | view in chronology ]
[ link to this | view in chronology ]