A large-scale collaborative assessment of the replicability and robustness of EEG research

脑电被广泛用于研究人类认知和其他心理现象。然而，尽管EEG结果很受欢迎，但它的可信度最近受到了争论。这种新的怀疑是基于这样一种观察，即新的假设通常只在小样本中得到检验，而重复研究通常被认为没有吸引力。此外，任何脑电分析都有很大的灵活性，因此分析管道在不同研究中是高度可变的。目前尚不清楚替代的、看似合理的管道在多大程度上会产生不同的结果和结论。我们认为，如果不通过使用新数据和替代分析来评估脑电研究的结果来评估其可重复性和稳健性，我们可能正在建造一座纸牌屋。受到心理科学复制危机的启发，我们有一个独特的机会，为脑电研究创造更坚实的基础。在这项提案中，我们提出了两个大型的国际合作项目，分别解决脑电研究的可复制性和稳健性，#EEGManyLabs将跨分散的实验室集中资源，并以前所未有的统计能力评估一些最有影响力的心理学发现的可复制性。#EEGManyLabs构成了迄今为止认知神经科学中最实质性的复制努力之一，并将为EEG研究留下持久的遗产。这些复制将更新我们对被广泛引用的现象的信心，并允许我们创建一个效应大小的库，这将为未来的实验设计提供信息。通过生成历史上最大的开放获取脑电数据集，未来的项目将能够更好地理解导致脑电结果变化的因素。最后，通过展示涉及大量脑电研究人员的大规模多地点项目的可行性，我们希望促进文化转变，从小规模的单一实验室实验转向高功率、社区驱动的合作。#EEGManyPipeline项目是一个国际多分析师项目，在该项目中，所有参与的研究人员都被提供相同的数据集，并被指示使用他们认为合理和代表他们自己研究的分析管道来分析数据。然后，分析人员将报告他们的结果和分析管道的详细描述，使我们能够分析分析管道的多样性及其对结果的影响。因此，这个项目将有助于通过替代分析评估EEG结果的稳健性，确定(次)最佳分析渠道，并在出版物中提供报告EEG分析的指导方针。我们预计这个项目将有助于提高EEG结果的可信度和分析质量，并将启发进行和报告EEG研究的新标准，从而支持未来EEG研究的基础。

EEG is widely used to investigate human cognition and other psychological phenomena. Yet, despite its popularity, the credibility of EEG findings has recently been debated. This new skepticism is based on the observation that novel hypotheses are oftentimes tested only in small samples, while replication studies are usually deemed unattractive. Moreover, there is a great deal of flexibility in any EEG analysis, such that analysis pipelines are highly variable across studies. The degree to which alternative, plausible pipelines yield different results and conclusions is currently unknown. We contend that without assessing the replicability and robustness of EEG research by assessing its results using new data and alternative analyses, we are potentially building a house of cards. Inspired by the lessons emerging from the replication crisis in the psychological sciences, we have a unique opportunity to create a stronger foundation for EEG research. In this proposal, we present two large-scale, international collaborative projects addressing the replicability and robustness of EEG research, respectively,#EEGManyLabs will pool resources across distributed laboratories and assess replicability of some of the most influential psychological findings with unprecedented statistical power. #EEGManyLabs constitutes one of the most substantial replication efforts in cognitive neuroscience to date, and will have a long-lasting legacy for EEG research. The replications will update our confidence in widely cited phenomena and allow us to create a library of effect sizes that will inform the design of future experiments. Through generating the largest collection of open access EEG datasets in history, future projects will be able to better understand the factors that contribute to variation in EEG findings. Finally, through demonstrating the feasibility of large-scale multi-site projects involving a large body of EEG researchers, we expect to facilitate a cultural shift away from small-scale single laboratory experiments towards high-powered, community driven collaborations.The #EEGManyPipelines project is an international many-analyst project, in which all participating researchers are provided with the same dataset and are instructed to analyze the data with an analysis pipeline they deem sensible and representative of their own research. Analysts will then report their results and a detailed description of the analysis pipeline, allowing us to analyze the diversity of analysis pipelines and their effects on results. Thus, this project will help assess the robustness of EEG findings across alternative analyses, identifying (sub)optimal analysis pipelines, and informing guidelines for reporting EEG analyses in publications.We expect that this project will help improve the credibility of EEG findings and the quality of analyses, and will inspire new standards for conducting and reporting EEG studies, thereby supporting the foundation of future EEG research.