A large-scale collaborative assessment of the replicability and robustness of EEG research

点击翻译按钮获取中文摘要

EEG is widely used to investigate human cognition and other psychological phenomena. Yet, despite its popularity, the credibility of EEG findings has recently been debated. This new skepticism is based on the observation that novel hypotheses are oftentimes tested only in small samples, while replication studies are usually deemed unattractive. Moreover, there is a great deal of flexibility in any EEG analysis, such that analysis pipelines are highly variable across studies. The degree to which alternative, plausible pipelines yield different results and conclusions is currently unknown. We contend that without assessing the replicability and robustness of EEG research by assessing its results using new data and alternative analyses, we are potentially building a house of cards. Inspired by the lessons emerging from the replication crisis in the psychological sciences, we have a unique opportunity to create a stronger foundation for EEG research. In this proposal, we present two large-scale, international collaborative projects addressing the replicability and robustness of EEG research, respectively,#EEGManyLabs will pool resources across distributed laboratories and assess replicability of some of the most influential psychological findings with unprecedented statistical power. #EEGManyLabs constitutes one of the most substantial replication efforts in cognitive neuroscience to date, and will have a long-lasting legacy for EEG research. The replications will update our confidence in widely cited phenomena and allow us to create a library of effect sizes that will inform the design of future experiments. Through generating the largest collection of open access EEG datasets in history, future projects will be able to better understand the factors that contribute to variation in EEG findings. Finally, through demonstrating the feasibility of large-scale multi-site projects involving a large body of EEG researchers, we expect to facilitate a cultural shift away from small-scale single laboratory experiments towards high-powered, community driven collaborations.The #EEGManyPipelines project is an international many-analyst project, in which all participating researchers are provided with the same dataset and are instructed to analyze the data with an analysis pipeline they deem sensible and representative of their own research. Analysts will then report their results and a detailed description of the analysis pipeline, allowing us to analyze the diversity of analysis pipelines and their effects on results. Thus, this project will help assess the robustness of EEG findings across alternative analyses, identifying (sub)optimal analysis pipelines, and informing guidelines for reporting EEG analyses in publications.We expect that this project will help improve the credibility of EEG findings and the quality of analyses, and will inspire new standards for conducting and reporting EEG studies, thereby supporting the foundation of future EEG research.