Eliminating the Invariance on the Loss Landscape of Linear Autoencoders
Eliminating the Invariance on the Loss Landscape of Linear Autoencoders
复制标题
DOI：
--
复制
发表时间：
2020-07
期刊：
影响因子：
--
通讯作者：
R. Oftadeh;Jiayi Shen;Zhangyang Wang;Dylan A. Shell
中科院分区：
文献类型：
--
作者：
R. Oftadeh;Jiayi Shen;Zhangyang Wang;Dylan A. Shell
本文提出了一种新的线性自编码器（LAE）的损失函数，并解析地量化了相关的损失表面的结构。优化传统的均方误差（MSE）损失导致跨越数据的样本协方差的主子空间的解码器矩阵，但是，由于在全局映射中抵消的不变性，它将无法识别精确的特征向量。我们在这里表明，我们提出的损失函数消除了这个问题，所以解码器收敛到样本协方差矩阵的精确有序非归一化特征向量。我们通过建立所有临界点的集合的解析表达式来表征新损失景观的完整结构，表明它是MSE的临界点的子集，并且所有局部极小值仍然是全局的。具体地说，MSE下的不变全局极小值在新损失下成为鞍点。此外，损失及其梯度的计算复杂度与MSE相同，因此，新损失不仅具有理论重要性，而且具有实用价值，例如，低秩近似
This paper proposes a new loss function for linear autoencoders (LAEs) and analytically iden-tifies the structure of the associated loss surface. Optimizing the conventional Mean Square Error (MSE) loss results in a decoder matrix that spans the principal subspace of the sample covariance of the data, but, owing to an invariance that cancels out in the global map, it will fail to identify the exact eigenvectors. We show here that our proposed loss function eliminates this issue, so the decoder converges to the exact ordered unnormal-ized eigenvectors of the sample covariance matrix. We characterize the full structure of the new loss landscape by establishing an analytical expression for the set of all critical points, showing that it is a subset of critical points of MSE, and that all local minima are still global. Specifically, the invariant global minima under MSE are shown to become saddle points under the new loss. Additionally, the computational complexity of the loss and its gradients are the same as MSE and, thus, the new loss is not only of theoretical importance but is of practical value, e.g., for low-rank approximation.