Randomized Automatic Differentiation
Randomized Automatic Differentiation
复制标题
DOI：
--
复制
发表时间：
2020-07
期刊：
影响因子：
--
通讯作者：
Deniz Oktay;N. McGreivy;Joshua Aduol;Alex Beatson;Ryan P. Adams
中科院分区：
文献类型：
--
作者：
Deniz Oktay;N. McGreivy;Joshua Aduol;Alex Beatson;Ryan P. Adams
深度学习、变分推理和许多其他领域的成功都得益于反向模式自动微分（AD）的专门实现，以计算兆维目标的梯度。这些工具的AD技术旨在将精确梯度计算到数值精度，但现代机器学习模型几乎总是使用随机梯度下降进行训练。为什么要在精确（小批量）梯度上花费计算和内存，而只将它们用于随机优化？我们开发了一个通用的框架和方法，随机自动微分（RAD），它允许无偏梯度估计计算减少内存，以换取方差。我们研究的一般方法的局限性，并认为，我们必须利用问题的具体结构，以实现利益。我们开发RAD技术的各种简单的神经网络架构，并表明，对于一个固定的内存预算，RAD收敛在更少的迭代比使用一个小批量的前馈网络，并在一个类似的数量为经常性的网络。我们还表明，RAD可以应用于科学计算，并使用它来开发一个低记忆随机梯度法优化控制参数的线性反应扩散偏微分方程代表裂变反应堆。
The successes of deep learning, variational inference, and many other fields have been aided by specialized implementations of reverse-mode automatic differentiation (AD) to compute gradients of mega-dimensional objectives. The AD techniques underlying these tools were designed to compute exact gradients to numerical precision, but modern machine learning models are almost always trained with stochastic gradient descent. Why spend computation and memory on exact (minibatch) gradients only to use them for stochastic optimization? We develop a general framework and approach for randomized automatic differentiation (RAD), which allows unbiased gradient estimates to be computed with reduced memory in return for variance. We examine limitations of the general approach, and argue that we must leverage problem specific structure to realize benefits. We develop RAD techniques for a variety of simple neural network architectures, and show that for a fixed memory budget, RAD converges in fewer iterations than using a small batch size for feedforward networks, and in a similar number for recurrent networks. We also show that RAD can be applied to scientific computing, and use it to develop a low-memory stochastic gradient method for optimizing the control parameters of a linear reaction-diffusion PDE representing a fission reactor.