Classification based hypothesis testing in neuroscience: Below-chance level classification rates and overlooked statistical properties of linear parametric classifiers
Classification based hypothesis testing in neuroscience: Below-chance level classification rates and overlooked statistical properties of linear parametric classifiers
复制标题
DOI：
10.1002/hbm.23140
复制
发表时间：
2016-05-01
影响因子：
4.8
通讯作者：
Gais, Steffen
中科院分区：
文献类型：
--
作者：
Jamalabadi, Hamidreza;Alizadeh, Sarah;Gais, Steffen
多元模式分析（MVPA）最近已成为数据分析的流行工具。通常，通过正确分类率 (CCR) 量化的分类准确度用于说明所研究的影响的大小。然而，我们表明，在神经科学中常见的低样本量 (LSS)、低效应量 (LES) 数据中，线性 MVPA 交叉验证的 CCR 分布是不对称的，并且可以显示分类率大大低于机会分类的预期。相反，这些情况下的分布模式高于预期的机会水平，导致高于机会 CCR 的数量虚假地高。当使用 MVPA 进行假设检验时，这种意外的分布具有很强的影响。我们的分析证明了这样的结论：CCR 不能很好地反映所调查的影响的大小。此外，零分布的偏度妨碍了使用许多标准参数检验来评估 CCR 的显着性。我们建议 MVPA 结果应以 P 值的形式报告，P 值是使用随机化测试估计的。此外，我们的结果表明，交叉验证程序使用少量的折叠，例如尽管平均 CCR 通常远低于使用更多折叠次数获得的结果，但双倍通常更敏感。 Hum Brain Mapp 37:1842-1855, 2016。(c) 2016 Wiley periodicals, Inc.
Multivariate pattern analysis (MVPA) has recently become a popular tool for data analysis. Often, classification accuracy as quantified by correct classification rate (CCR) is used to illustrate the size of the effect under investigation. However, we show that in low sample size (LSS), low effect size (LES) data, which is typical in neuroscience, the distribution of CCRs from cross-validation of linear MVPA is asymmetric and can show classification rates considerably below what would be expected from chance classification. Conversely, the mode of the distribution in these cases is above expected chance levels, leading to a spuriously high number of above chance CCRs. This unexpected distribution has strong implications when using MVPA for hypothesis testing. Our analyses warrant the conclusion that CCRs do not well reflect the size of the effect under investigation. Moreover, the skewness of the null-distribution precludes the use of many standard parametric tests to assess significance of CCRs. We propose that MVPA results should be reported in terms of P values, which are estimated using randomization tests. Also, our results show that cross-validation procedures using a low number of folds, e.g. twofold, are generally more sensitive, even though the average CCRs are often considerably lower than those obtained using a higher number of folds. Hum Brain Mapp 37:1842-1855, 2016. (c) 2016 Wiley Periodicals, Inc.