A Simple and Efficient Tensor Calculus
A Simple and Efficient Tensor Calculus
复制标题
DOI：
10.1609/aaai.v34i04.5881
复制
发表时间：
2020-04
期刊：
影响因子：
--
通讯作者：
S. Laue;Matthias Mitterreiter;Joachim Giesen
中科院分区：
文献类型：
--
作者：
S. Laue;Matthias Mitterreiter;Joachim Giesen
计算张量表达式的导数，也称为张量演算，是机器学习中的一项基本任务。一个关键的问题是评估表达式及其衍生物的效率，这取决于这些表达式的表示。最近，已经引入了一种用于计算张量表达式（如雅可比矩阵或海森矩阵）的高阶导数的算法，该算法比先前的最先进的方法快几个数量级。不幸的是，该方法基于Ricci符号，因此无法纳入使用更简单的Einstein符号的自动微分框架，如TensorFlow，PyTorch，autograd或JAX。这留下了两个选择，要么改变这些框架中的底层张量表示，要么开发一个新的，基于爱因斯坦符号的可证明正确的算法。显然，第一种选择是不切实际的。因此，我们采用第二种选择。在这里，我们表明，使用Ricci符号是没有必要的一个有效的张量演算和开发一个同样有效的方法，更简单的爱因斯坦符号。事实证明，转向爱因斯坦符号可以进一步改进，从而提高效率。
Computing derivatives of tensor expressions, also known as tensor calculus, is a fundamental task in machine learning. A key concern is the efficiency of evaluating the expressions and their derivatives that hinges on the representation of these expressions. Recently, an algorithm for computing higher order derivatives of tensor expressions like Jacobians or Hessians has been introduced that is a few orders of magnitude faster than previous state-of-the-art approaches. Unfortunately, the approach is based on Ricci notation and hence cannot be incorporated into automatic differentiation frameworks like TensorFlow, PyTorch, autograd, or JAX that use the simpler Einstein notation. This leaves two options, to either change the underlying tensor representation in these frameworks or to develop a new, provably correct algorithm based on Einstein notation. Obviously, the first option is impractical. Hence, we pursue the second option. Here, we show that using Ricci notation is not necessary for an efficient tensor calculus and develop an equally efficient method for the simpler Einstein notation. It turns out that turning to Einstein notation enables further improvements that lead to even better efficiency.