Exceeding chance level by chance: The caveat of theoretical chance levels in brain signal classification and statistical assessment of decoding accuracy
Exceeding chance level by chance: The caveat of theoretical chance levels in brain signal classification and statistical assessment of decoding accuracy
复制标题
DOI：
10.1016/j.jneumeth.2015.01.010
复制
发表时间：
2015-07-30
影响因子：
3
通讯作者：
Jerbi, Karim
中科院分区：
文献类型：
--
作者：
Combrisson, Etienne;Jerbi, Karim
机器学习技术越来越多地用于神经科学中对大脑信号进行分类。解码性能通过分类结果与纯随机分类所实现的速率的偏离程度来反映。在2类或4类分类问题中，概率水平分别为50%或25%。然而，这样的阈值适用于无限数量的数据样本，但不适用于小数据集。虽然这种限制在机器学习领域得到了广泛的认可，但不幸的是，在新兴的大脑信号分类领域，它有时仍然被忽视或忽略。顺便说一句，这个领域经常面临着低样本量的困难。在这项研究中，我们演示了如何将信号分类高斯随机信号可以产生高达70%或更高的解码精度在两类解码小样本集。最重要的是，我们提供了一个彻底的量化的严重程度和参数影响这种限制使用模拟，我们操纵样本量，类数，交叉验证参数（k倍，留一和重复次数）和分类器类型（线性判别分析，朴素贝叶斯和支持向量机）。除了发出警告的红旗外，我们还说明了分析和经验解决方案（二项式公式和排列测试）的使用，这些解决方案通过提供解码准确性的统计显著性水平（p值）来解决问题，同时考虑到样本量。最后，我们通过评估脑磁图（MEG）和颅内EEG（iEEG）基线记录中的噪声水平分类来说明我们对真实的大脑数据的模拟和统计测试的相关性。(C)2015 Elsevier B.V.版权所有。
Machine learning techniques are increasingly used in neuroscience to classify brain signals. Decoding performance is reflected by how much the classification results depart from the rate achieved by purely random classification. In a 2-class or 4-class classification problem, the chance levels are thus 50% or 25% respectively. However, such thresholds hold for an infinite number of data samples but not for small data sets. While this limitation is widely recognized in the machine learning field, it is unfortunately sometimes still overlooked or ignored in the emerging field of brain signal classification. Incidentally, this field is often faced with the difficulty of low sample size. In this study we demonstrate how applying signal classification to Gaussian random signals can yield decoding accuracies of up to 70% or higher in two-class decoding with small sample sets. Most importantly, we provide a thorough quantification of the severity and the parameters affecting this limitation using simulations in which we manipulate sample size, class number, cross-validation parameters (k-fold, leave-one-out and repetition number) and classifier type (Linear-Discriminant Analysis, Naive Bayesian and Support Vector Machine). In addition to raising a red flag of caution, we illustrate the use of analytical and empirical solutions (binomial formula and permutation tests) that tackle the problem by providing statistical significance levels (p-values) for the decoding accuracy, taking sample size into account. Finally, we illustrate the relevance of our simulations and statistical tests on real brain data by assessing noise-level classifications in Magnetoencephalography (MEG) and intracranial EEG (iEEG) baseline recordings. (C) 2015 Elsevier B.V. All rights reserved.