If science journalists can report results discussed at conferences, why should preprints be off-limit, especially if reported responsibly? While peer-reviewed papers published in reputable journals give journalists a degree of confidence in reporting, there have been instances when such papers have caused much harm. Peer-reviewing is not a magic bullet as the number of papers with duplicated images show. Responsible journalism is what matters more.
Extraordinary claims require extraordinary evidence. And to broaden the aphorism popularised by Carl Sagan, I would say extraordinary claims (made by scientists) invite extraordinary scrutiny by peers and coverage by science journalists. To think otherwise is naivety.
The reason for writing this post is to clear some misconceptions that lay people in general and scientists in particular have about science journalists writing articles about preprints deposited in repositories. The context for this is the heated discussion on social media following media coverage of a preprint deposited in arXiv by a two-member team of Dev Kumar Thapa and Prof. Anshu Pandey from Indian Institute of Science (IISc), Bengaluru. The team had apparently observed nanosized films and pellets made of silver nanoparticles embedded in a gold matrix exhibiting superconductivity at ambient temperature (-37 degree C) and pressure.
The post I wrote (for want of space and other constraints a shorter version was published in The Hindu) based on the preprint did not elicit any comments from any scientist on why I chose to write on a preprint and not wait for the paper to be peer-reviewed and published in Nature, where the authors had submitted their manuscript. Since the authors were prohibited from speaking to the media, I spoke to a few physicists working in the field of superconductivity to deconstruct the paper and make the findings of the paper accessible to lay readers. I cross-checked certain details with Prof. Arindam Ghosh at IISc before publishing the piece. There was not a tinge of hype or hyperbole in the article.
The next article that I wrote on the subject was after Brian Skinner, a physicist at the Massachusetts Institute of Technology, posted his comments on the arXiv repository. He raised a red flag after noticing nearly identical noise in two presumably independent measurements of the magnetic susceptibility in Prof. Pandey’s preprint. Noise, by its very virtue, should and will be random and finding nearly identical noise in measurements made under different conditions is therefore highly improbable.
From arXiv to social media
Incidentally, Skinner did not stop with posting his comment on arXiv repository. The scientific discussion, nay monologue, spilled on to social media when starting August 9, in a series of tweets, Skinner explained the potential problem with the IISc study. By the time I wrote the second article on August 12, Skinner had got a reply from IISc researchers. “Thanks for pointing this out! We hadn’t noticed this peculiar noise correlation. We don’t know its origin yet,” they had said in an email to him. Skinner tweeted this as well and went one step further by saying: “I’ve had another email exchange with the authors, and I will just say: They are REALLY not backing down from their claims. They emphasize that they are focused on providing validation of their data, and will only post new data or a response to my note once they have done so.”
As if Skinner taking to social media to dissect what he felt was a problem with the preprint, which should have ideally be restricted to comments section, was not enough, the IISc researchers did not feel compelled to restrict themselves to arXiv repository. They instead emailed him. If the purpose was to keep the message private, then Skinner defeated it by tweeting the crux of the message.
And Prof. Pratap Raychaudhuri from the Superconductivity Lab at Tata Institute of Fundamental Research (TIFR) Mumbai too took to Facebook (on August 12 and hours before I posted my article) to explain what he felt was an alternative possibility for the origin of identical noise.
The entire discussion was in public domain but not a single soul found it odd. But some scientists suddenly and strongly suddenly felt I should have waited for the paper to be published before writing the article. One scientist (I don’t want to divulge the name) even when to the extent of saying I should have waited for the authors to speak to me after the embargo ends before posting Skinner’s views, which were anyway tweeted days ago.
To these scientists who voiced their opinion and to others who felt the same but chose to remain silent let me explain how science journalism works. While published papers, especially those published in high impact journals (I am aware of the problems with impact factor), are the main source based on which science journalists across the world write articles, they are not the only source.
Scientific conferences are another major source of information that many science journalists routinely bank on. And many of the presentations at these conferences would not have even reached the stage of submission to a peer-reviewed journal but are worthy enough to be written about. If there is no problem with articles written based on information gathered during conferences (and obviously cross checked with a few experts), why is there such commotion when it comes to reporting on preprints? Especially when they are done responsibly and are not on health- and medicine-related issues, which might cause immediate danger if the findings reported are downright wrong or cause unintended harm?
Journalists do have greater degree of confidence while reporting on a study published in a peer-reviewed journal, especially the ones which have high impact factor. But there are umpteen number of cases where science journalists have ended up communicating very wrong message by relying on peer-reviewed papers.
Tom Sheldon’s article in Nature News, has in so many words said that preprints in the hands of science journalists can end up writing articles that are plain wrong or “misleading”. And one of the hypothetical cases he cites is how early findings that show a common vaccine is unsafe could cause much harm. I wonder how he chose to cite this example and how it passed scrutiny.
The MMR-autism association
Has Sheldon ever heard of the now infamous study published in 1998 by Dr. Andrew Wakefield and his colleagues in one of the most respected, high-impact, peer-reviewed journal called The Lancet? In short, the study was about the association of MMR vaccine with autism in eight of the 12 children reported. “Many parents seeking a cause for their children’s illness seized upon the apparent link between the routine vaccination and autism,” says a 2010 paper in The Canadian Medical Association Journal. This study resulted in a precipitous drop in the number children being vaccinated leading to “dramatic health consequences”. There were large measles outbreaks in 2008 and 2009 in the UK following a drop in vaccination. The paper was completely retracted only after 12 long years and after BMJ wrote a series of articles exposing the fraud.
GM corn and cancer
Sheldon then refers to Gilles-EricSéralini’s paper on how rats fed GM corn developed cancer to drive home the point of how “public understanding has been distorted by media coverage of ambiguous or just downright bad science”. Though journalists were not allowed to cross check the findings with scientists not connected with the paper, Sheldon seems to have forgotten that the paper was indeed published in a peer-reviewed journal.
Scientists of all hue and colour know that peer-reviewing is far from perfect. Retraction Watch blog and websites such a Pubpeer, a website that allows users researchers to discuss and review scientific papers that have been published, will cease to exist if peer-reviewing was only good enough.
Publication of a paper in a journal cannot be considered final. For all the peer-reviewing that has gone into a manuscript before acceptance and publication, there are umpteen papers published in top-notch peer-reviewed journals that do not stand further study. First is the ability to reproduce the experiment by the same group and then by other independent research groups across the world. There is now “growing alarm about results that cannot be reproduced”, says Nature.
A survey carried out by Nature found that there a reproducibility crisis in science. As much as two-thirds of scientists who responded to Nature survey admitted that reproducibility is a major problem. “Pressure to publish, selective reporting, poor use of statistics and finicky protocols can all contribute to wobbly work,” says the Nature editorial.
Peer-reviewed papers with duplicated images
A day before Sheldon published his News piece, a paper by Dr. Elisabeth M. Bik and others published in Molecular and Cellular Biology found as many as 35,000 papers published in peer-reviewed journals between 2009 and 2015 are candidates for retraction due to image duplication. Sheldon would have had enough time to write a better informed piece had he only referred her preprint posted in the bioRxiv repository on June 24 (one month before he published his News item.) And a 2006 study by Dr. Bik and her colleagues found that looking at the country of origin of the authors of the 348 papers published in PLOS ONE that had duplicated images, China and India had a higher proportion of papers containing problematic images. But more number of papers from China and the U.S. contain duplicated images.
In an interview, Dr. Bik told me: “Most [of the duplicated images] are easy to spot for me, but apparently not for others. All published papers have gone through peer review and editorial handling, and papers I am scanning have been published months or years ago, so there were several opportunities for others to see them.”
In all fairness, the near-identical noise in IISc team’s result could have been spotted by Nature peer-reviewers and the authors alerted. But looking at the numerous instances where even simple image duplication within the same panel in a paper has been missed by peer-reviewers and other scientists reading the paper, I doubt if peer-reviewers would noticed that. Skinner had to magnify or zoom (as he refers to) the bottom part of the figure to see the near-identical noise in two independent measurements of the magnetic susceptibility. Would peer-reviewers have done that?
Evidence nearer home
The proof that peer-reviewers have less ability to spot even simple image duplication can be seen in two recent cases in India where researchers from IIT Dhanbad and Bose Institute, Kolkata have been found to be serial offenders. Fourteen papers have been retracted and two corrected for image duplication and another two-dozen problematic papers are still listed on Pubpeer in the case of IIT Dhanbad researchers. The Bose Institute case is quite similar — two papers have been retracted and two corrected and about a dozen more problematic papers are still listed on Pubpeer. There are scientists from other Indian labs who have fewer problematic papers listed on Pubpeer website.
In the last one month, almost 90% of papers from Indian labs (including CSIR labs) and nodal institutions in the country posted on Pubpeer website have issues of image duplication. These papers have been published in peer-reviewed journals (some being really reputable ones).
Since the papers have been listed only in the last one month none of them have been retracted or corrected. The main reason why I spent a few days scanning Pubpeer website was to know which scientists’ work, even if they are from reputed institutions and published in respected journals, I should avoid reporting, lest I end up promoting fake science.
When peer-reviewing could have helped
But yes, there have been cases where the main result reported in preprints have proved to be completely wrong. In September 2011, the OPERA team posted a preprint in arXiv claiming that neutrinos travelled 60 nanoseconds faster than light. It later became apparent that the calculations were wrong, and the mistake was due to faulty wiring.
And in March 2014, scientists at the Harvard-Smithsonian Centre for Astrophysics in the U.S announced the indirect detection of gravitational waves in the afterglow of the Big Bang only to be proved wrong.
These two announcements prior to peer-reviewing have caused a lot of embarrassment and two members of the OPERA team had to step down owing to severe criticism. It is premature to say anything about the IISc superconductivity study.
“They [IISc team] are REALLY not backing down from their claims. They emphasize that they are focused on providing validation of their data, and will only post new data or a response to my note once they have done so,” Skinner tweeted. “People at IISc are being tight-lipped, and the official statement from the authors is that they’re waiting to “have their data validated by another group” before they reply.”
Unlike religion which is dogmatic, this episode gives a non-scientist outsider a peep at how science self-corrects. And that perhaps is the correct way to look at it without getting emotional.