Mad Max: Affine Spline Insights Into Deep Learning
Mad Max: Affine Spline Insights Into Deep Learning
复制标题
DOI：
10.1109/jproc.2020.3042100
复制
发表时间：
2018-05
影响因子：
20.6
通讯作者：
Randall Balestriero;Richard Baraniuk
中科院分区：
文献类型：
--
作者：
Randall Balestriero;Richard Baraniuk
我们通过样条函数和操作员在深网（DNS）和近似理论之间建立了严格的桥梁。我们的关键结果是，可以将大量的DNS写作作为Max-Affine样条算子（MASOS）的组成，它们提供了一个强大的门户网站，我们可以通过该门户来查看和分析其内部工作。例如，在包含输入信号的条件分区区域的条件下，MASO DN的输出可以写成输入的简单仿射转换。这意味着DN构造了一组信号依赖性的，类特异性的模板，通过简单的内部产物比较信号。我们通过匹配的过滤器和数据记忆的影响探讨了与最佳分类的经典理论的链接。进一步，我们提出了一个简单的罚款项，可以将其添加到任何DN学习算法的成本函数中，以迫使模板彼此正交。这导致分类的性能显着提高，并减少过度拟合，而无需更改DN体系结构。 MASO隐式诱导的输入信号空间的样条分区直接将DNS与向量量化理论（VQ）和K-Means聚类联系起来，这为研究DNS如何以层次结构方式组织信号。为了验证VQ解释的效用，我们为信号和图像开发并验证一个新的距离度量，以量化其VQ编码之间的差异。
We build a rigorous bridge between deep networks (DNs) and approximation theory via spline functions and operators. Our key result is that a large class of DNs can be written as a composition of max-affine spline operators (MASOs) that provide a powerful portal through which we view and analyze their inner workings. For instance, conditioned on the spline partition region containing the input signal, the output of an MASO DN can be written as a simple affine transformation of the input. This implies that a DN constructs a set of signal-dependent, class-specific templates against which the signal is compared via a simple inner product; we explore the links to the classical theory of optimal classification via matched filters and the effects of data memorization. Going further, we propose a simple penalty term that can be added to the cost function of any DN learning algorithm to force the templates to be orthogonal with each other; this leads to significantly improved classification performance and reduced overfitting with no change to the DN architecture. The spline partition of the input signal space that is implicitly induced by an MASO directly links DNs to the theory of vector quantization (VQ) and K-means clustering, which opens up new geometric avenues to study how DNs organize signals in a hierarchical fashion. To validate the utility of the VQ interpretation, we develop and validate a new distance metric for signals and images that quantify the difference between their VQ encodings.