International effort tackles reproduction of original science results

The most comprehensive investigation ever made into the reproducibility of results in a field of science has found that, regardless of the analytic method or criteria used, fewer than half of the replications produced the same findings as the original study.

The massive international collaboration involved 270 researchers, who published their findings in Science today. They conducted replications of 100 studies published in three prominent psychology journals.

“Our findings suggest that, as a field, we have some work ahead of us. But by identifying some key predictors of reproducibility, and by suggesting ways to increase it, the project marks an important milestone in improving our research practices,” said Dr Patrick Goodbourn, a researcher in the School of Psychology at the University of Sydney.

The University of Sydney and the Australian National University are the only two Australian institutions to have taken part in the study, which involved 125 institutions in 17 countries.

Launched in 2012, the Reproducibility Project: Psychology, co-ordinated by the Centre for Open Science in Virginia, USA, tests scientists’ reliance on the reproducibility of results to gain confidence in ideas and evidence. 

Reproducibility means that the results recur when the same data are analysed again, or when new data are collected using the same methods.

Fewer than half of the original findings tested were successfully replicated. This held true across multiple different criteria of success.

However, the team is careful to point out that a failure to replicate doesn’t always mean that the original finding is false. Some replications will simply have failed to detect the original result by chance. And even though most replication teams worked with the original authors to use the same materials and methods, small differences in when, where, or how the replication was carried out might influence the outcome.

The report’s authors also recognise that a problem for psychology and other fields is that incentives for scientists are not always aligned with reproducibility.

“What is good for science and what is good for scientists are not always the same thing. In the present culture, scientists’ key incentive is earning publications of their research, particularly in prestigious outlets,” said Ljiljana Lazarević, a team member from the University of Belgrade.

Research with new, surprising findings is more likely to be published than research examining when, why, or how existing findings can be reproduced.

“Poor reproducibility has been suggested anecdotally, and has been argued from a theoretical standpoint, but now we have solid evidence that our research practices don’t always generate reproducible findings,” said Dr Goodbourn. 

Suggestions for addressing the issue include improving access to all aspects of the original research and the pre-registration of research designs. In keeping with the goals of openness and reproducibility, every replication project undertaken in this study posted its methods, data and computer code on a public website.

Many organisations are already working on the issue of reproducibility, including the Reproducibility Project: Cancer Biology.

“The great thing about science when it is operating properly is that it can reflect on its own practices and correct its mistakes. This project is an example of science doing what it does best,” said Dr Goodbourn.