The Perils and Pitfalls of Block Design for EEG Classification Experiments
The Perils and Pitfalls of Block Design for EEG Classification Experiments
复制标题
DOI：
10.1109/tpami.2020.2973153
复制
发表时间：
2021-01-01
影响因子：
23.6
通讯作者：
Siskind, Jeffrey Mark
中科院分区：
文献类型：
--
作者：
Li, Ren;Johansen, Jared S.;Siskind, Jeffrey Mark
最近的一篇论文[1]声称，通过脑电图（EEG）对观看ImageNet刺激的受试者所引起的大脑处理进行分类，并利用从该处理中得到的表征来构建一个新的对象分类器。这篇论文，以及随后的一系列论文[2]，[3],[4],[5],[6],[7],[8]，声称在各种计算机视觉任务上取得了成功的结果，包括对象分类，迁移学习，以及使用脑电图测量的脑源性表征生成描绘人类感知和思维的图像。我们的新实验和分析表明，他们的结果在很大程度上取决于他们采用的块设计，其中所有给定类别的刺激都在一起呈现，而在快速事件设计中失败，其中不同类别的刺激随机混合。区块设计导致基于已知存在于所有EEG数据中的区块级时间相关性，而不是刺激相关活动，对任意大脑状态进行分类。由于他们的测试集中的每个试验与相应训练集中的许多试验来自同一块，因此他们的块设计导致对数据的任意时间工件进行分类，而不是对刺激相关的活动进行分类。这使得在多篇发表的论文中对该数据进行的所有后续分析无效，并对所有报告的结果提出质疑。我们进一步表明，使用随机码本构建的新对象分类器的性能与使用从EEG数据中提取的表示构建的新对象分类器一样好，甚至更好，这表明使用从EEG数据中提取的表示构建的分类器的性能并没有从脑源表示中受益。总之，我们的结果说明了存在于所有神经成像数据中的时间自相关性对分类实验的深远影响。此外，我们的结果校准了所涉及的任务的潜在难度，并告诫人们不要过于乐观，但不正确，相反的说法。
A recent paper [1] claims to classify brain processing evoked in subjects watching ImageNet stimuli as measured with EEG and to employ a representation derived from this processing to construct a novel object classifier. That paper, together with a series of subsequent papers [2], [3], [4], [5], [6], [7], [8], claims to achieve successful results on a wide variety of computer-vision tasks, including object classification, transfer learning, and generation of images depicting human perception and thought using brain-derived representations measured through EEG. Our novel experiments and analyses demonstrate that their results crucially depend on the block design that they employ, where all stimuli of a given class are presented together, and fail with a rapid-event design, where stimuli of different classes are randomly intermixed. The block design leads to classification of arbitrary brain states based on block-level temporal correlations that are known to exist in all EEG data, rather than stimulus-related activity. Because every trial in their test sets comes from the same block as many trials in the corresponding training sets, their block design thus leads to classifying arbitrary temporal artifacts of the data instead of stimulus-related activity. This invalidates all subsequent analyses performed on this data in multiple published papers and calls into question all of the reported results. We further show that a novel object classifier constructed with a random codebook performs as well as or better than a novel object classifier constructed with the representation extracted from EEG data, suggesting that the performance of their classifier constructed with a representation extracted from EEG data does not benefit from the brain-derived representation. Together, our results illustrate the far-reaching implications of the temporal autocorrelations that exist in all neuroimaging data for classification experiments. Further, our results calibrate the underlying difficulty of the tasks involved and caution against overly optimistic, but incorrect, claims to the contrary.