Variability in the analysis of an event-related potential dataset by many teams: Pruning the garden of forking paths in EEG research

我们最近提出，在人格神经科学中，导致可复制率不令人满意的主要驱动因素是统计能力低、理论基础薄弱和数据分析过于灵活。在我们正在进行的项目中，我们通过将协作方法应用到研究中来解决这些问题，从严格审查的假设到收集独特的大型和丰富的EEG数据集。虽然所有成员不断为项目目标作出贡献，但持续的大流行病使收集数据的时间超过12个月，从而大大推迟了项目的完成。除了成功完成正在进行的项目外，我们还计划扩大对项目更新中分析灵活性问题的调查。为了防止选择导致预期结果的分析路径(即P-hack)，建议预先登记假设和分析。然而，替代分析路径的存在引发了关于尝试复制的成功(或失败)的问题，因为做出这些选择的客观标准通常不可用。因此，整个分析工作流程的预注册需要做出不确定但有影响的分析决策。最近，盲数据分析(即，根据要分析的全部数据开发分析工作流程，只排除兴趣的影响)被建议作为一种替代方法。然而，将复杂的分析策略调整到完整但盲目的数据集的机会是否会带来明显更好的分析选择，仍有待检验。如果是这样的话，这将对当前的开放科学建议产生直接影响，即在数据收集之前预先注册分析管道。此外，即使在以前对许多独立分析员进行比较的少数几项研究中，目前也不清楚分析选择的可变性是否受到研究人员专业知识/经验或他们对影响的信念的可变性的影响。因此，在这个项目中，我们将以事件相关电位(ERPs)和人格特征之间的关联研究为例，(1)识别和评估复杂脑电数据集的分析选择的影响，(2)评估盲目分析是否有助于改进分析选择，以及(3)检查研究人员的经验/专业知识是否有助于分析选择和结果的变异性。为此，我们将邀请大量分析师提交分析路径，以在大型数据集上测试特定的ERP个性关联。通过将这些路径与文献和我们当前的CoScience项目进行比较，我们期望为如何做出分析决策提供切实可行的建议，这可以为脑电研究的其他子领域以及邻近领域提供蓝图。

We have recently suggested that in personality neuroscience, the main drivers for unsatisfactory low replicability rates are low statistical power, a weak theoretical base and too much flexibility in data analysis. In our ongoing project we address these problems by applying a collaborative approach to research, spanning from critically reviewed hypotheses to the collection of a uniquely large and rich EEG dataset. While all members continuously contributed to the project goals, the ongoing pandemic precluded data collection for more than 12 months thereby significantly delaying project completion. Besides successfully finishing the ongoing project, we plan to expand our investigation of the problem of analytical flexibility in a project renewal. To prevent the selection of an analysis path that results in a desired outcome (i.e., P-HACKING), the preregistration of hypotheses and analyses has been recommended. However, the existence of alternative analysis paths raises questions regarding the success (or failure) of an attempted replication, because objective criteria for making these choices are often not available. Preregistration of the entire analysis workflow therefore requires making underdetermined but consequential analysis decisions. Recently, blind data analysis (i.e., developing the analysis workflow based on the full data to be analysed only excluding the effect of interest) has been suggested as an alternative approach. However, it remains to be tested whether the opportunity to adapt a complex analysis strategy to the full, yet blinded, data set results in measurably better analysis choices. If this were the case, this would have direct implications for current open science recommendations to preregister analysis pipelines before data collection. Furthermore, even in the few previous studies that compared many independent analysts, it is currently unclear whether variability of analysis choices was affected by variability in either researchers’ expertise/experience or in their beliefs concerning the effect. In this project, we will therefore use research on associations between event-related potentials (ERPs) and personality traits as an example to (1) identify and evaluate the impact of analysis choices of a complex EEG data set, (2) evaluate whether blind analysis can help improving analysis choices, and (3) examine whether researcher experience/expertise contributes to variability in analysis choices and results. We will do so by inviting a large number of analysts to submit analysis paths for testing specific ERP-personality associations on a large data set. By comparing these with paths based on the literature and our current CoScience project, we expect to provide practical recommendations on how to make analysis decisions, which can provide a blueprint for other subfields in EEG research as well as adjacent fields.