Forward- or Reverse-Mode Automatic Differentiation: What's the Difference?
Forward- or Reverse-Mode Automatic Differentiation: What's the Difference?
复制标题
DOI：
10.48550/arxiv.2212.11088
复制
发表时间：
2022-12
期刊：
影响因子：
--
通讯作者：
Birthe van den Berg;Tom Schrijvers;James McKinna;Alexander Vandenbroucke
中科院分区：
文献类型：
--
作者：
Birthe van den Berg;Tom Schrijvers;James McKinna;Alexander Vandenbroucke
自动微分（AD）一直是许多学科研究人员感兴趣的话题，自从其应用于机器学习和神经网络以来，越来越受欢迎。尽管许多研究人员欣赏并知道如何应用AD，但真正了解其潜在过程仍然是一个挑战。然而，从代数的观点来看，AD似乎是惊人的自然：它源于微分定律。在这项工作中，我们使用编程代数技术来推理不同的AD变体，利用Haskell来说明我们的观察结果。我们的发现源于三个基本的代数抽象：（1）半环上的模的概念，（2）Nagata的“模的理想化”的构造，以及（3）Kronecker的δ函数，它们一起允许我们编写AD的单行抽象定义。从这个单线定义，并通过实例化我们的代数结构以各种方式，我们得到不同的AD变体，具有相同的扩展行为，但不同的内涵属性，主要是在（渐近）计算复杂性方面。我们通过Kronecker同构展示了不同的变体，进一步阐述了我们的Haskell基础设施，通过构建保证了正确性。有了这个框架，本文试图使广告变种更容易理解，采取代数的角度来看这个问题。
Automatic differentiation (AD) has been a topic of interest for researchers in many disciplines, with increased popularity since its application to machine learning and neural networks. Although many researchers appreciate and know how to apply AD, it remains a challenge to truly understand the underlying processes. From an algebraic point of view, however, AD appears surprisingly natural: it originates from the differentiation laws. In this work we use Algebra of Programming techniques to reason about different AD variants, leveraging Haskell to illustrate our observations. Our findings stem from three fundamental algebraic abstractions: (1) the notion of module over a semiring, (2) Nagata's construction of the 'idealization of a module', and (3) Kronecker's delta function, that together allow us to write a single-line abstract definition of AD. From this single-line definition, and by instantiating our algebraic structures in various ways, we derive different AD variants, that have the same extensional behaviour, but different intensional properties, mainly in terms of (asymptotic) computational complexity. We show the different variants equivalent by means of Kronecker isomorphisms, a further elaboration of our Haskell infrastructure which guarantees correctness by construction. With this framework in place, this paper seeks to make AD variants more comprehensible, taking an algebraic perspective on the matter.