In an important new paper, Rafael Ahlskog and Sven Oskarsson show that half of the effect size in observational studies, even in conservative estimates, is composed of confounding. In laypeople’s terms, this means that many academic studies considerably overestimate the effect of the variables which they study. Such results erode confidence in academic work, yet there have been a series of these findings. In a 2018 paper, 29 expert teams used a single dataset to analyze whether football referees were more likely to red card dark-skin-toned players, the teams coming to radically different results, none of which could be proved incorrect. Such findings follow the replication crisis. In 2015, researchers replicated 100 leading studies in the field of psychology, finding that only 36% of the replications yielded significant findings. Moreover, the mean effect size in the replications was about half the effect reported in the original studies. In other fields, similar crises followed.
Within academia, such developments are well known. Yet public awareness is limited and few reflect on practical consequences for politics. Though such challenges do not affect academic confidence in the reality of (say) climate change and Coronavirus, there are significant consequences for policies which address such issues; if scholars worry about findings, how can we be confident that policy interventions are effective? Admittedly, political implications are indirect. Decades ago, postmodern philosophers deconstructed social reality, engendering loss of confidence in liberal democracy among certain intellectuals. Yet this had questionable impact on practical politics. At this time, politicians such as Ronald Reagan and George W. Bush promoted liberal democracy with an evangelical zeal, reflecting a popular culture in which confidence in democracy remained overwhelming. But recent trends are distinct. Quantitative social scientists now worry about the validity of methods, previous crises of confidence being confined to the humanities.