The Role Robust Statistical Methods in the Credibility Revolution of Psychological Science

点击翻译按钮获取中文摘要

The credibility movement in psychology arose as a response to failures to replicate a numberof effects that were previously considered robust (Open Science Collaboration, 2015).Previous research indicated that the flexibility the researchers exercise when deciding howto analyse their data, which transformation to apply, and which cases to retain or exclude(Steegen et al., 2016), has contributed current replicability issues. Analytic decision makingcan be influenced by a number of factors, including bias towards producing results that aremore likely to get published because of statistical significance, but may subsequently fail toreplicate. This project will investigate how limiting analytic flexibility in the research process- namely by preregistering analytic plans and applying robust statistical methods thatremain unbiased under conditions where other commonly used models underperform - canhelp address some of the problems with replicability in psychology by reducing thepossibilities for generating false-positive effects and improving power to discover effectsthat would otherwise remain undetected. It will do so by: (1) querying psychologyresearchers about their knowledge and understanding of the assumptions of the generallinear model (GLM), the most commonly applied statistical model in psychology, and aboutthe solutions they apply to counteract bias from commonly occurring characteristics of datafrom psychological research; (2) using Natural Language Processing (NLP) to text-mineinformation from published papers and examine the language the researchers use todescribe their data, statistical models and draw conclusions about the effects underinvestigation; (3) assessing the performance of robust statistical methods under real-worldconditions of psychological data (e.g., typical distributional characteristics, presence ofoutliers and influential cases, etc.); (4) evaluating the robustness of published effects forpreregistered and non-preregistered studies. The project will shine a light on currentstatistical practice within psychological research and lead to specific 'best practice'recommendations for psychology researchers at all career stages. This work will address theemerging methodological challenges in the areas of Open Science, replicability, andtransparency of research practice, thus having a direct impact on the improvement ofpsychology as a scientific discipline.