ResNet Autoencoders for Unsupervised Feature Learning From High-Dimensional Data: Deep Models Resistant to Performance Degradation
ResNet Autoencoders for Unsupervised Feature Learning From High-Dimensional Data: Deep Models Resistant to Performance Degradation
复制标题
DOI：
10.1109/access.2021.3064819
复制
发表时间：
2021-01-01
期刊：
影响因子：
3.9
通讯作者：
Manic, Milos
中科院分区：
文献类型：
--
作者：
Wickramasinghe, Chathurika S.;Marino, Daniel L.;Manic, Milos
高维数据的高效建模需要通过特征学习仅提取相关维度。无监督特征学习由于其无偏的方法，不需要先验知识或昂贵的手动处理，以及处理指数数据增长的能力而获得了极大的关注。Deep Autoencoder（AE）是一种用于无监督特征学习的最先进的深度神经网络，它使用一系列堆叠层来学习嵌入式表示。然而，随着AE网络变得更深，这些学习的嵌入式表示可能会由于梯度消失而恶化，导致性能下降。本文介绍了用于无监督特征学习的ResNet Autoencoder（RAE）及其卷积版本（C-RAE）。RAE和C-RAE的优点是，与标准AE相比，它使用户能够添加剩余连接以增加网络容量，而不会导致无监督特征学习的降级成本。虽然RAE和C-RAE继承了AE的所有优点，例如自动非线性特征提取和无监督学习，但它们也允许用户设计更大的网络，而不会对特征学习性能产生不利影响。我们对学习的嵌入式表示进行分类，以评估RAE和C-RAE。将RAE和C-RAE与MNIST、Fashion MNIST和CIFAR 10数据集上的AE进行比较。当增加层数时，C-RAE的表现优于AE，因为与AE（33%至65%）相比，C-RAE的分类准确性性能下降（小于3%）显着降低。此外，C-RAE表现出更高的平均准确度和更低的准确度比标准AE的方差。当RAE和C-RAE与广泛使用的特征学习方法（卷积AE，PCA，伊卡，LLE，因子分析和SVD）进行比较时，C-RAE显示出最高的准确性。
Efficient modeling of high-dimensional data requires extracting only relevant dimensions through feature learning. Unsupervised feature learning has gained tremendous attention due to its unbiased approach, no need for prior knowledge or expensive manual processing, and ability to handle exponential data growth. Deep Autoencoder (AE) is a state-of-the-art deep neural network for unsupervised feature learning, which learns embedded-representations using a series of stacked layers. However, as the AE network gets deeper, these learned embedded-representations can deteriorate due to vanishing gradient, leading to performance degradation. This article presents ResNet Autoencoder (RAE) and its convolutional version (C-RAE) for unsupervised feature learning. The advantage of RAE and C-RAE is that it enables the user to add residual connections for increased network capacity without incurring the cost of degradation for unsupervised feature learning compared to standard AEs. While RAE and C-RAE inherit all the advantages of AEs, such as automated non-linear feature extraction and unsupervised learning, they also allow users to design larger networks without adverse effects on feature learning performance. We performed classification on learned embedded-representation to evaluate RAE and C-RAE. RAE and C-RAE were compared against AEs on MNIST, Fashion MNIST, and CIFAR10 datasets. When increasing the number of layers, C-RAE outperformed AE by showing significantly lower performance degradation of classification accuracy (less than 3%) compared to AE (33% to 65%). Further, C-RAE exhibited higher mean accuracy and lower variance of accuracy than standard AE. When comparing RAE and C-RAE with widely used feature learning methods (Convolutional AE, PCA, ICA, LLE, Factor Analysis, and SVD), C-RAE showed the highest accuracy.