If you want to believe that men are so sexist they will die rather than flee from a hurricane with a female name, you can find a study to show that. If you want to believe white people think black people look blacker during bad economic times, Facebook is controlling your emotions or if you are bad at gambling it is genetic, an academic paper can make your case.
And not just in any obscure journal. Those papers were each published in one of the most respected peer-reviewed journals in the world, the Proceedings of the National Academy of Sciences (PNAS) and all just this year.
If those papers can make it into PNAS, imagine the quality of the papers in some of the lesser known of the 25,000 journals that now exist. They all claim to be peer-reviewed but we know that can’t be true. There are about 600,000 practicing scientists in the US. With 25,000 journals today, even if they each only published 2 articles per month it would mean every scientist in the U.S. had to be spending their time doing peer review for free. And those 25,000 journals publish a lot more than 2 each; PLOS One alone publishes 30,000 articles each year.
We could dismiss those articles as outliers, every journal has had bad articles slip through, and in lesser-known publications they would get little notice, but all of those articles got mainstream media coverage because they had the stamp of PNAS. PNAS brings legitimacy because the public and journalists do not realize that PNAS standards have been lax for some time. A body once so elite even the legendary Carl Sagan, he of Cosmos fame, did not qualify, has long let members hand-walk papers through peer review for friends of theirs.
In the Wall Street Journal, I detailed one alarming failure of their peer review process; in both 2002 and 2010, it published papers claiming that frogs were being damaged by a common herbicide named atrazine, and because of the credibility of the journal the findings were published in America’s premier newspapers, causing alarm throughout the country. In both instances the Environmental Protection Agency launched investigations into the product.
Then, in both instances, after spending years of time and millions of dollars in taxpayer money redoing work already done re-registering atrazine the EPA, which is certainly not regarded as friendly to business much less chemical companies, cleared it.
From the Wall Street Journal article
In 2002 and 2010, papers published in the Proceedings of the National Academy of Sciences claimed that a pesticide called atrazine was causing sex changes in frogs. As a result the Environmental Protection Agency set up special panels to re-examine the product’s safety. Both papers had the same editor, David Wake of the University of California, Berkeley, who is a colleague of the papers’ lead author, Tyrone Hayes, also of Berkeley.
In keeping with National Academy of Sciences policy, Prof. Hayes preselected Prof. Wake as his editor. Both studies were published without a review of the data used to reach the finding. No one has been able to reproduce the results of either paper, including the EPA, which did expensive, time-consuming reviews of the pesticide brought about by the published claims. As the agency investigated, it couldn’t even use those papers about atrazine’s alleged effects because the research they were based on didn’t meet the criteria for legitimate scientific work. The authors refused to hand over data that led them to their claimed results—which meant no one could run the same computer program and match their results.
What went wrong? The EPA had decided to examine the hypothesis based on the credibility of the journal without considering the validity of the paper. Once they began studying the matter, they were unable to even use the papers that had gotten their regulatory and investigational juggernaut rolling – because the papers included no data and the methodology did not meet the standards of the EPA for what a well-controlled, prospective study looks like. The EPA also did not realize that the papers had been edited by a person hand-picked for the job before the paper was even submitted, an Academy member who happened to be a friend of the senior author, and in the same department at the University of California, Berkeley.
The author who benefited from being walked through peer review by a friend in the Academy, Professor Tyrone Hayes, is now sponsored on trips by groups like the Pesticide Action Network but he will have a more difficult time getting into PNAS; after my article in the Wall Street Journal and the ensuing controversy, the Academy announced it was canceling its ‘prearranged editor’ policy.
There is hope for the future. Dr. Francis Collins, director of the National Institutes of Health oversees $29 billion in academic funding, and he has made it clear he is concerned about similar ‘secret sauce’ methods and lack of data that means lack of reproducibility. PLOS One, the most prolific article publisher in the world, has announced they are going to require that data be included.
That’s bad for fuzzy claims based on anecdotes, surveys of college students or agenda-based efforts, but it is good for science. Because even if “peer reviewed” now commonly means ‘read by an editor’ rather than criticized by experts, as it used to mean, the inclusion of data makes it possible for scientists to do post-publication peer review. Post-publication debunking and retractions of papers won’t make their way into prominent newspapers the way outrageous claims do, but if mainstream media doesn’t stop writing articles based on the name of the journal, the public will lose confidence in them.
While that gets sorted out, as long as less-than-reputable scientists are no longer able to game the peer review system, and mainstream media gets back to being trusted guides for the public on complex science issues, that is a win for all of us.
Hank Campbell is founder of Science 2.0 and an award-winning science writer who has appeared in numerous publications, from Wired to the Wall Street Journal. In 2012 he was co-author of the bestselling book Science Left Behind. Follow him on Twitter @HankCampbell.
