When a massive replicability study in psychology was published last year, the results were, to some, shocking: 60% of the 100 experimental results failed to replicate. Now, the latest attempt to verify findings in the social sciences—this time with a small batch from experimental economics—also finds a substantial number of failed replications. Following the exact same protocols of the original studies, the researchers failed to reproduce the results in about 40% of cases.
“I find it reassuring that the replication rate was fairly high,” says Michael L. Anderson, an economist at the University of California, Berkeley, not involved with the study. But he notes that most of the failures came from studies using a 5% “p value” cut-off for statistical significance, suggesting “what some realize but fewer are willing to discuss: The accepted standard of a 5% significance level is not sufficient to generate results that are likely to replicate.”
Psychology’s high-profile replication efforts, which were cautiously welcomed by the research community, have triggered policy changes at some scientific journals and modified priorities at many funding agencies. But the overall failure rate has also been called into question, because most of the original studies were reproduced only once, often without strictly following the initial protocol. And most of the replication studies allowed the replicators to choose their targets.