The replication crisis: Researchers reveal a hidden universe of uncertainty

The University of Luxembourg’s Department of Social Sciences contributed to a large-scale replication study that aimed to understand the role of decisions that scientists make during the research process. Published in the Proceedings of the National Academy of Sciences, the study highlights the importance of open science and collaboration among scientists.

During the COVID 19 pandemic, a certain crisis of credibility in science manifested itself. It has also become apparent that both the public and experts have difficulty dealing with scientific uncertainty.

Part of this mix is the “replication crisis”: In several scientific fields, studies have repeatedly failed to replicate a significant proportion of previously published results. Even if this can be explained by errors, publication pressure and bias or even questionable research practices, the failure of replications undermines the role of science as a reliable producer of knowledge.

A study led by Nate Breznau and colleagues investigated the role of decisions that scientists make during the research process. Andreas Hadjar and Robin Samuel from the Department of Social Sciences at the Faculty of Humanities, Education and Social Sciences of the University of Luxembourg contributed to this large-scale replication study, which involved 161 researchers in 73 teams from around the world.

One hypothesis, one data set, various results

The focus of the replication was to test the hypothesis that more immigration reduces public support for government provision of social measures.

Interestingly, the 1,253 statistical models contributed by the research teams produced very different results—even though they used the same data. The analytical choices, the expertise of the researchers and their expectations could not explain these huge differences in the results.

The authors concluded that a “hidden universe of uncertainty remains.” Consequently, “scientists, especially those working with the complexities of human societies and behavior, should exercise humility and strive to better account for the uncertainty in their work.” The authors further highlight the potential and importance of transparent and collaborative research.