Are open set classification methods effective on large-scale datasets?
Are open set classification methods effective on large-scale datasets?
复制标题
DOI：
10.1371/journal.pone.0238302
复制
发表时间：
2020-09
期刊：
影响因子：
3.7
通讯作者：
Ryne Roady;Tyler L. Hayes;Ronald Kemker;Ayesha Gonzales;Christopher Kanan
中科院分区：
文献类型：
--
作者：
Ryne Roady;Tyler L. Hayes;Ronald Kemker;Ayesha Gonzales;Christopher Kanan
监督分类方法通常假设训练数据和测试数据分布相同，并且测试集中的所有类都存在于训练集中。然而，部署的分类器通常需要能够将训练集外部的输入识别为未知数。这个问题已经在多种范式下进行了研究，包括分布外检测和开放集识别。对于卷积神经网络，有两种主要方法：1）将已知与未知分开的推理方法；2）特征空间正则化策略以提高模型对新输入的鲁棒性。到目前为止，很少有人关注探索两种方法之间的关系以及直接比较具有数十个类别的大规模数据集的性能。使用 ImageNet ILSVRC-2012 大规模分类数据集，我们确定了正则化和专门推理方法的新颖组合，这些方法在难度级别不断增加的多个开放集分类问题上表现最佳。我们发现，无论特征空间正则化策略如何，输入扰动和温度缩放在大规模数据集上都比其他测试的推理方法产生更好的性能。相反，我们发现，在使用基线推理技术时，在训练期间通过高级正则化方案提高性能会产生更好的性能；然而，当使用先进的推理方法来检测开放集类时，这些繁琐的训练范例的效用就不那么明显了。
Supervised classification methods often assume the train and test data distributions are the same and that all classes in the test set are present in the training set. However, deployed classifiers often require the ability to recognize inputs from outside the training set as unknowns. This problem has been studied under multiple paradigms including out-of-distribution detection and open set recognition. For convolutional neural networks, there have been two major approaches: 1) inference methods to separate knowns from unknowns and 2) feature space regularization strategies to improve model robustness to novel inputs. Up to this point, there has been little attention to exploring the relationship between the two approaches and directly comparing performance on large-scale datasets that have more than a few dozen categories. Using the ImageNet ILSVRC-2012 large-scale classification dataset, we identify novel combinations of regularization and specialized inference methods that perform best across multiple open set classification problems of increasing difficulty level. We find that input perturbation and temperature scaling yield significantly better performance on large-scale datasets than other inference methods tested, regardless of the feature space regularization strategy. Conversely, we find that improving performance with advanced regularization schemes during training yields better performance when baseline inference techniques are used; however, when advanced inference methods are used to detect open set classes, the utility of these combersome training paradigms is less evident.