slimTrain---A Stochastic Approximation Method for Training Separable Deep Neural Networks
slimTrain---A Stochastic Approximation Method for Training Separable Deep Neural Networks
复制标题
slimTrain---一种训练可分离深度神经网络的随机逼近方法
DOI：
10.1137/21m1452512
复制
发表时间：
2022
影响因子：
3.1
通讯作者：
Ruthotto, Lars
中科院分区：
文献类型：
--
作者：
Newman, Elizabeth;Chung, Julianne;Chung, Matthias;Ruthotto, Lars
深度神经网络（dnn）作为高维函数逼近器在许多应用中取得了成功；然而，训练dnn通常是具有挑战性的。DNN训练通常被描述为一个随机优化问题，其挑战包括非凸性、非光滑性、正则化不足和复杂的数据分布。因此，dnn在给定任务上的性能主要取决于超参数的调整，特别是学习率和正则化参数。在缺乏类似任务的理论指导或先前经验的情况下，这需要解决一系列重复的训练问题，这可能是耗时且需要计算资源的。这可能会限制深度神经网络在非标准、复杂和稀缺数据集问题上的适用性，例如，在许多科学应用中出现的问题。为了解决深度神经网络训练的挑战，我们提出了\tt slimTrain，这是一种训练深度神经网络的随机优化方法，对超参数选择的敏感性降低，初始收敛速度快。slimTrain的核心思想是利用许多深度神经网络架构固有的可分离性；也就是说，我们将深度神经网络分成一个非线性特征提取器，然后是一个线性模型。这种可分性使我们能够利用最近在解决大规模、线性、不适定逆问题方面取得的进展。至关重要的是，对于线性权值，\tt slimTrain不需要学习率并自动适应正则化参数。在我们的数值实验中，使用代理建模和降维中产生的函数近似任务，slimTrain优于现有的推荐超参数设置的DNN训练方法，并降低了DNN训练对剩余超参数的敏感性。由于我们的方法在小批量上操作，每次迭代的计算开销是适度的，并且可以通过减少迭代次数（由于更快的初始收敛）或需要解决的训练问题的数量来实现节省，以识别有效的超参数。
Deep neural networks (DNNs) have shown their success as high-dimensional function approximators in many applications; however, training DNNs can be challenging in general. DNN training is commonly phrased as a stochastic optimization problem whose challenges include nonconvexity, nonsmoothness, insufficient regularization, and complicated data distributions. Hence, the performance of DNNs on a given task depends crucially on tuning hyperparameters, especially learning rates and regularization parameters. In the absence of theoretical guidelines or prior experience on similar tasks, this requires solving a series of repeated training problems which can be time-consuming and demanding on computational resources. This can limit the applicability of DNNs to problems with nonstandard, complex, and scarce datasets, e.g., those arising in many scientific applications. To remedy the challenges of DNN training, we propose \tt slimTrain, a stochastic optimization method for training DNNs with reduced sensitivity to the choice of hyperparameters and fast initial convergence. The central idea of slimTrain is to exploit the separability inherent in many DNN architectures; that is, we separate the DNN into a nonlinear feature extractor followed by a linear model. This separability allows us to leverage recent advances made for solving large-scale, linear, ill-posed inverse problems. Crucially, for the linear weights, \tt slimTrain does not require a learning rate and automatically adapts the regularization parameter. In our numerical experiments using function approximation tasks arising in surrogate modeling and dimensionality reduction, slimTrain outperforms existing DNN training methods with the recommended hyperparameter settings and reduces the sensitivity of DNN training to the remaining hyperparameters. Since our method operates on mini-batches, its computational overhead per iteration is modest and savings can be realized by reducing the number of iterations (due to quicker initial convergence) or the number of training problems that need to be solved to identify effective hyperparameters.