Don't take advice from me cause I'm MEAN

Open Data
Posted by Steven Novella on February 26, 2014 (42 Comments)

PLOS (the Public Library of Science) is a non-profit open access publisher of science articles. Their goal is to make scientific data accessible to everyone, in the name of transparency and open communication. Now they have taken their approach one step further, announcing their policy that all articles published in a PLOS journal must submit their original data so that anyone can access and analyze it for themselves.

In an effort to increase access to this data, we are now revising our data-sharing policy for all PLOS journals: authors must make all data publicly available, without restriction, immediately upon publication of the article. Beginning March 3rd, 2014, all authors who submit to a PLOS journal will be asked to provide a Data Availability Statement, describing where and how others can access each dataset that underlies the findings. This Data Availability Statement will be published on the first page of each article.

They allow for exceptions—when subject confidentiality is an issue, sensitive information related to endangered species, and when the authors do not own the data. In such cases, however, data must be available upon request, and not controlled by the authors. Otherwise the raw data must be made available.

I think this is a fabulous idea, for many reasons. We frequently write here at SBM about the challenges faced by the various institutions of science to maintain high standards of quality and transparency. Those challenges include publication bias, the literature being flooded with preliminary or low quality research, researchers exploiting degrees of freedom (also referred to as “p-hacking”) without their questionable behavior being apparent in the final published paper, conflicts of interest, the relative lack of replications and lack of desire on the part of editors to publish replications, frequent statistical errors and the occasional deliberate fraud.

There are many ways to erode the quality of scientific research, or to manipulate research to achieve a desired end (rather than discover what is real). In the end, however, I am not nihilistic. Science can and does still move forward, although slowly. We just have to recognize how messy the process is so that we can best sift out the noise and find the reliable evidence.

It does feel as if we are in an era of self-examination and increased efforts to identify and correct the failings of modern scientific research. It further seems as if the transparency and immediate access to data afforded by the internet is largely responsible for this. This is also an era of experimentation where various models are being proposed or tried as potential solutions to the various problems faced by science.

The Open Access movement is one such experiment, and PLOS has been its flagship. In my opinion, the experiment has been a partial success, but has created some of its own problems. It is certainly extremely useful to have immediate access to a full published article when researching a topic. This facilitates post-publication peer review, and the discussion within the community about the research. PLOS has also managed to maintain a reasonably high quality among its journals.

However, open access journals without such high standards have also proliferated. The business model of most open access journals is that they do not have subscriptions (by definition) so they pay for themselves by charging authors a publication fee. This can create a perverse incentive to publish lots of low quality papers in order to garner those fees, and since publication is only online (without the expense of print journals), creating minimalist open access journals with terrible quality control can be profitable. Last year Science published the results of a “sting” operation exposing the poor quality of many open access journals (PLOS, to its credit, did not fall for the sting).

Open access is therefore not a panacea, and comes with its own challenges. It can, however, address the issues of transparency and universal access to facilitate review and discussion. I therefore think it is a great idea for PLOS to go “all in” on this strategy. If you are going for transparency, then make the raw data transparent, not just the final worked-over data.

In fact we have proposed this previously as one strategy to combat the problem of p-hacking. If researchers disclose the process by which they collected data, all the data they collected, and every way it was analyzed, then p-hacking would become more transparent, and this would hopefully discourage the practice. At the very least it would make it easier for other researchers to reanalyze the data to see if the results are genuine or an artifact of creative analysis.

In fact, I think all journals should adopt this policy. Researchers should make available to journal editors all their raw data, so that the journal can make it available either online or on request to other researchers who want to review or reanalyze the data, or just to help them replicate the study.

Conclusion

I think we are in an exciting time in the evolution of the institutions of science. Many problems that have been festering for a long time are being exposed and discussed. This may be unsettling—to learn about all the flaws in the practice of science. But these flaws all have potential solutions, and many of them are not difficult at all, they just need the will to execute.

Journals and their editors are largely the gatekeepers for the official record of scientific research—the published peer-reviewed literature. Therefore many of the solutions to these problems rest with them. Open access is one approach, that I feel will have a long future and play an important role in reforming the institutions of science. Requiring open access to data is a great move that capitalizes on the strength of the open access movement.

Print journal editors, in fact, would be well-advised to follow suit.

In recent years it has become policy for researchers to disclose funding and potential conflicts of interest. Clinical trial registries have also been created so that companies cannot hide research whose results they don’t like. But more reforms still are needed.

Universities should institute more uniform education of researchers so that they are more aware of the problems of p-hacking, to minimize error and bias and to maximize scientific rigor.

Journal editors should publish more negative studies and more replications (including exact replications). Not everything has to end up as a full article in the print version of the journal. Online supplements can provide the space needed to publish whatever is necessary to maximize the unbiased flow of quality scientific information.

Science is often characterized as a self-corrective process. The process of science itself needs to be self-corrective. These fixes do not require massive resources, just reasonable changes in policy. PLOS should be commended for helping lead the way, at least with respect to open access to data.

People should learn to agree to disagree.