The Role Robust Statistical Methods in the Credibility Revolution of Psychological Science

心理学中的可信度运动是对未能复制一些以前被认为是稳健的影响的反应(Open Science Collaboration，2015)。先前的研究表明，研究人员在决定如何分析他们的数据、应用哪些转换以及保留或排除哪些案例时所行使的灵活性，导致了当前的可复制性问题(Steegen等人，2016)。分析性决策可能会受到许多因素的影响，包括偏向于产生更有可能因统计学意义而公布但随后可能无法复制的结果。这个项目将调查在研究过程中限制分析灵活性--即通过预先登记分析计划和应用稳健的统计方法--在其他常用模型表现不佳的情况下保持公正--如何通过减少产生假阳性效应的可能性和提高发现否则仍未被发现的效果的能力，来帮助解决心理学中的一些可复制性问题。它将通过：(1)询问心理学研究人员关于他们对心理学中最常用的统计模型--一般线性模型(GLM)的假设的知识和理解，以及他们用来抵消来自心理学研究的常见数据特征的偏差的解决方案；(2)使用自然语言处理(NLP)从已发表的论文中挖掘信息，并检查研究人员用来描述其数据、统计模型并得出关于被调查影响的语言的语言；(3)评估稳健的统计方法在心理数据的现实条件下的表现(例如，典型的分布特征、离群点的存在和有影响力的案例等)；(4)评估注册前和非注册前研究发表的效果的稳健性。该项目将揭示心理学研究中当前的统计实践，并为心理学研究人员在所有职业生涯阶段提供具体的“最佳实践”建议。这项工作将解决开放科学、可复制性和研究实践透明度等领域新出现的方法论挑战，从而对心理学作为一门科学学科的发展产生直接影响。

The credibility movement in psychology arose as a response to failures to replicate a numberof effects that were previously considered robust (Open Science Collaboration, 2015).Previous research indicated that the flexibility the researchers exercise when deciding howto analyse their data, which transformation to apply, and which cases to retain or exclude(Steegen et al., 2016), has contributed current replicability issues. Analytic decision makingcan be influenced by a number of factors, including bias towards producing results that aremore likely to get published because of statistical significance, but may subsequently fail toreplicate. This project will investigate how limiting analytic flexibility in the research process- namely by preregistering analytic plans and applying robust statistical methods thatremain unbiased under conditions where other commonly used models underperform - canhelp address some of the problems with replicability in psychology by reducing thepossibilities for generating false-positive effects and improving power to discover effectsthat would otherwise remain undetected. It will do so by: (1) querying psychologyresearchers about their knowledge and understanding of the assumptions of the generallinear model (GLM), the most commonly applied statistical model in psychology, and aboutthe solutions they apply to counteract bias from commonly occurring characteristics of datafrom psychological research; (2) using Natural Language Processing (NLP) to text-mineinformation from published papers and examine the language the researchers use todescribe their data, statistical models and draw conclusions about the effects underinvestigation; (3) assessing the performance of robust statistical methods under real-worldconditions of psychological data (e.g., typical distributional characteristics, presence ofoutliers and influential cases, etc.); (4) evaluating the robustness of published effects forpreregistered and non-preregistered studies. The project will shine a light on currentstatistical practice within psychological research and lead to specific 'best practice'recommendations for psychology researchers at all career stages. This work will address theemerging methodological challenges in the areas of Open Science, replicability, andtransparency of research practice, thus having a direct impact on the improvement ofpsychology as a scientific discipline.