Variability in the analysis of an event-related potential dataset by many teams: Pruning the garden of forking paths in EEG research

我们最近提出，在人格神经科学中，导致低复制率的主要驱动因素是统计能力低、理论基础薄弱和数据分析灵活性太大。在我们正在进行的项目中，我们通过应用协作方法进行研究来解决这些问题，从严格审查的假设到收集独特的大型和丰富的EEG数据集。虽然所有成员都在不断地为项目目标作出贡献，但持续的大流行病使数据收集工作中断了12个多月，从而大大推迟了项目的完成。除了成功地完成正在进行的项目，我们计划扩大我们的调查的问题，分析的灵活性，在一个项目更新。为了防止选择导致期望结果的分析路径（即，P-HACKING），建议预先登记假设和分析。然而，替代分析路径的存在引起了关于尝试复制的成功（或失败）的问题，因为做出这些选择的客观标准通常不可用。因此，整个分析工作流程的预配准需要做出欠确定但重要的分析决策。最近，盲数据分析（即，建议采用一种替代办法，即根据要分析的全部数据制定分析工作流程，但不包括利益影响。然而，将复杂的分析策略应用于完整但设盲的数据集是否会导致可测量的更好的分析选择，仍有待检验。如果是这样的话，这将对当前开放科学建议在数据收集之前预注册分析管道产生直接影响。此外，即使在以前的几项研究中，比较了许多独立的分析师，目前还不清楚是否变异的分析选择的影响，无论是在研究人员的专业知识/经验或在他们的信念有关的影响。因此，在本项目中，我们将以事件相关电位（ERP）和人格特质之间的关联研究为例，（1）识别和评估复杂EEG数据集的分析选择的影响，（2）评估盲分析是否有助于改善分析选择，（3）检查研究人员的经验/专业知识是否有助于分析选择和结果的可变性。为此，我们将邀请大量分析师提交分析路径，以在大型数据集上测试特定的ERP-个性关联。通过与基于文献和我们当前的CoScience项目的路径进行比较，我们希望就如何做出分析决策提供实用的建议，这可以为EEG研究中的其他子领域以及相邻领域提供蓝图。

We have recently suggested that in personality neuroscience, the main drivers for unsatisfactory low replicability rates are low statistical power, a weak theoretical base and too much flexibility in data analysis. In our ongoing project we address these problems by applying a collaborative approach to research, spanning from critically reviewed hypotheses to the collection of a uniquely large and rich EEG dataset. While all members continuously contributed to the project goals, the ongoing pandemic precluded data collection for more than 12 months thereby significantly delaying project completion. Besides successfully finishing the ongoing project, we plan to expand our investigation of the problem of analytical flexibility in a project renewal. To prevent the selection of an analysis path that results in a desired outcome (i.e., P-HACKING), the preregistration of hypotheses and analyses has been recommended. However, the existence of alternative analysis paths raises questions regarding the success (or failure) of an attempted replication, because objective criteria for making these choices are often not available. Preregistration of the entire analysis workflow therefore requires making underdetermined but consequential analysis decisions. Recently, blind data analysis (i.e., developing the analysis workflow based on the full data to be analysed only excluding the effect of interest) has been suggested as an alternative approach. However, it remains to be tested whether the opportunity to adapt a complex analysis strategy to the full, yet blinded, data set results in measurably better analysis choices. If this were the case, this would have direct implications for current open science recommendations to preregister analysis pipelines before data collection. Furthermore, even in the few previous studies that compared many independent analysts, it is currently unclear whether variability of analysis choices was affected by variability in either researchers’ expertise/experience or in their beliefs concerning the effect. In this project, we will therefore use research on associations between event-related potentials (ERPs) and personality traits as an example to (1) identify and evaluate the impact of analysis choices of a complex EEG data set, (2) evaluate whether blind analysis can help improving analysis choices, and (3) examine whether researcher experience/expertise contributes to variability in analysis choices and results. We will do so by inviting a large number of analysts to submit analysis paths for testing specific ERP-personality associations on a large data set. By comparing these with paths based on the literature and our current CoScience project, we expect to provide practical recommendations on how to make analysis decisions, which can provide a blueprint for other subfields in EEG research as well as adjacent fields.