False-Positive Citations

Joseph P. Simmons, Leif D. Nelson, Uri Simonsohn

In 2010 or thereabouts, we stopped believing that many published findings were true. We discussed recently published articles in our weekly journal clubs (we were all at different universities then), and those discussions frequently devolved into statements of disbelief. We didn’t think the findings were fraudulent, but it was just impossible to believe that, with only 14 participants per cell, researchers had found that people will pay more for a chocolate bar when it is presented at a 45 degree angle, but only if they are below the median on the self-monitoring scale.1 When results in the scientific literature disagree with our intuition, we should be able to trust the literature enough to question our beliefs rather than to question the findings. We were questioning the findings. Something was broken.
After much discussion, our best guess was that so many published findings were false because researchers were conducting many analyses on the same dataset and just reporting those that were statistically significant, a behavior that we later labeled “p-hacking” (Simonsohn, Nelson, & Simmons, 2014).