Variability in the analysis of an event-related potential dataset by many teams: Pruning the garden of forking paths in EEG research

点击翻译按钮获取中文摘要

We have recently suggested that in personality neuroscience, the main drivers for unsatisfactory low replicability rates are low statistical power, a weak theoretical base and too much flexibility in data analysis. In our ongoing project we address these problems by applying a collaborative approach to research, spanning from critically reviewed hypotheses to the collection of a uniquely large and rich EEG dataset. While all members continuously contributed to the project goals, the ongoing pandemic precluded data collection for more than 12 months thereby significantly delaying project completion. Besides successfully finishing the ongoing project, we plan to expand our investigation of the problem of analytical flexibility in a project renewal. To prevent the selection of an analysis path that results in a desired outcome (i.e., P-HACKING), the preregistration of hypotheses and analyses has been recommended. However, the existence of alternative analysis paths raises questions regarding the success (or failure) of an attempted replication, because objective criteria for making these choices are often not available. Preregistration of the entire analysis workflow therefore requires making underdetermined but consequential analysis decisions. Recently, blind data analysis (i.e., developing the analysis workflow based on the full data to be analysed only excluding the effect of interest) has been suggested as an alternative approach. However, it remains to be tested whether the opportunity to adapt a complex analysis strategy to the full, yet blinded, data set results in measurably better analysis choices. If this were the case, this would have direct implications for current open science recommendations to preregister analysis pipelines before data collection. Furthermore, even in the few previous studies that compared many independent analysts, it is currently unclear whether variability of analysis choices was affected by variability in either researchers’ expertise/experience or in their beliefs concerning the effect. In this project, we will therefore use research on associations between event-related potentials (ERPs) and personality traits as an example to (1) identify and evaluate the impact of analysis choices of a complex EEG data set, (2) evaluate whether blind analysis can help improving analysis choices, and (3) examine whether researcher experience/expertise contributes to variability in analysis choices and results. We will do so by inviting a large number of analysts to submit analysis paths for testing specific ERP-personality associations on a large data set. By comparing these with paths based on the literature and our current CoScience project, we expect to provide practical recommendations on how to make analysis decisions, which can provide a blueprint for other subfields in EEG research as well as adjacent fields.