A large-scale collaborative assessment of the replicability and robustness of EEG research

脑电被广泛用于研究人类认知和其他心理现象。然而，尽管EEG研究结果很受欢迎，但其可信度最近一直存在争议。这种新的怀疑是基于这样的观察：新的假设通常只在小样本中进行测试，而复制研究通常被认为是没有吸引力的。此外，在任何EEG分析中存在很大的灵活性，使得分析管道在研究中高度可变。目前尚不清楚替代的、合理的管道产生不同结果和结论的程度。我们认为，如果不通过使用新数据和替代分析评估EEG研究的结果来评估其可复制性和鲁棒性，我们可能会建立一个纸牌屋。从心理科学的复制危机中吸取的教训启发，我们有一个独特的机会为脑电图研究奠定更坚实的基础。在这个提案中，我们提出了两个大规模的国际合作项目，分别解决EEG研究的可复制性和鲁棒性，#EEGManyLabs将汇集分布式实验室的资源，并以前所未有的统计能力评估一些最有影响力的心理学发现的可复制性。#EEGManyLabs是迄今为止认知神经科学领域最重要的复制工作之一，将为EEG研究带来持久的遗产。这些重复将更新我们对广泛引用的现象的信心，并使我们能够创建一个效应大小库，为未来的实验设计提供信息。通过生成历史上最大的开放获取EEG数据集，未来的项目将能够更好地了解导致EEG结果变化的因素。最后，通过展示涉及大量EEG研究人员的大规模多站点项目的可行性，我们希望促进从小规模单一实验室实验向高功率，社区驱动的合作的文化转变。#EEGManyPipelines项目是一个国际多分析师项目，其中，所有参与的研究人员都被提供了相同的数据集，并被指示使用他们认为合理且代表他们自己研究的分析管道来分析数据。然后，分析师将报告他们的结果和分析管道的详细描述，使我们能够分析分析管道的多样性及其对结果的影响。因此，该项目将有助于评估EEG结果在替代分析中的稳健性，确定（次）最佳分析管道，并为在出版物中报告EEG分析提供指导方针。我们预计，该项目将有助于提高EEG结果的可信度和分析质量，并将激发开展和报告EEG研究的新标准，从而为未来EEG研究奠定基础。

EEG is widely used to investigate human cognition and other psychological phenomena. Yet, despite its popularity, the credibility of EEG findings has recently been debated. This new skepticism is based on the observation that novel hypotheses are oftentimes tested only in small samples, while replication studies are usually deemed unattractive. Moreover, there is a great deal of flexibility in any EEG analysis, such that analysis pipelines are highly variable across studies. The degree to which alternative, plausible pipelines yield different results and conclusions is currently unknown. We contend that without assessing the replicability and robustness of EEG research by assessing its results using new data and alternative analyses, we are potentially building a house of cards. Inspired by the lessons emerging from the replication crisis in the psychological sciences, we have a unique opportunity to create a stronger foundation for EEG research. In this proposal, we present two large-scale, international collaborative projects addressing the replicability and robustness of EEG research, respectively,#EEGManyLabs will pool resources across distributed laboratories and assess replicability of some of the most influential psychological findings with unprecedented statistical power. #EEGManyLabs constitutes one of the most substantial replication efforts in cognitive neuroscience to date, and will have a long-lasting legacy for EEG research. The replications will update our confidence in widely cited phenomena and allow us to create a library of effect sizes that will inform the design of future experiments. Through generating the largest collection of open access EEG datasets in history, future projects will be able to better understand the factors that contribute to variation in EEG findings. Finally, through demonstrating the feasibility of large-scale multi-site projects involving a large body of EEG researchers, we expect to facilitate a cultural shift away from small-scale single laboratory experiments towards high-powered, community driven collaborations.The #EEGManyPipelines project is an international many-analyst project, in which all participating researchers are provided with the same dataset and are instructed to analyze the data with an analysis pipeline they deem sensible and representative of their own research. Analysts will then report their results and a detailed description of the analysis pipeline, allowing us to analyze the diversity of analysis pipelines and their effects on results. Thus, this project will help assess the robustness of EEG findings across alternative analyses, identifying (sub)optimal analysis pipelines, and informing guidelines for reporting EEG analyses in publications.We expect that this project will help improve the credibility of EEG findings and the quality of analyses, and will inspire new standards for conducting and reporting EEG studies, thereby supporting the foundation of future EEG research.