Validated computation of patterns in recurrent neural networks

循环神经网络中模式的验证计算

• 批准号: 493789610
• 负责人: Dr. Elena Queirolo, Ph.D.
• 金额: --
• 依托单位国家: 德国
• 项目类别: WBP Position
• 资助国家: 德国
• 项目状态: 未结题
• 来源: https://gepris.dfg.de/gepris/projekt/493789610?language=en
• 关键词: Validated; computation; patterns; recurrent; neural

The goal of this project is to employ validated computer-assisted techniques from dynamical systems to understand the behaviour of recurrent neural networks (RNNs). RNNs are one key component in recent machine learning algorithms, particularly in the context of deep neural networks. RNNs can be interpreted as dynamical systems on graphs, or networks. Nodes and edges are associated to state values and weights respectively. First, information processing by the nodes of a neural net with fixed link weights is a dynamical process as initial conditions from the input layer are processed and yield, via a finite-time iteration of maps or a finite-time flow, the values at the output layer. Second, also the learning phase of a neural network can be interpreted as a dynamical minimization problem obtained by iteration. Of course, there is a feedback between the dynamics on and of the network, i.e., on the nodes and of the links. At the end of the dynamical process obtained by processing a sufficient amount of training data, one hopes that the dynamical system has produced a sufficiently stable pattern of the link weights, which has sufficient expressive power for tasks such as pattern matching or even extrapolation beyond initial training data.We are interested in studying the stable network configuration, that is the pattern, that the weights will achieve. Such patterns are of great interest because they indicate when the RNN received sufficient training, and if the training objective has been achieved. This study would also be used to compare flows with different starting conditions and flows in RNNs with different architectures. This would allow us to choose the best combination of RNN and system architecture for the application under consideration.Applying concepts from validated numerics, it will be possible to find patterns and prove their existence and stability. This is going to allow us to mathematically determine their occurrence in a given RNN. Hence, this project centers around the following question:Can we rigorously predict the pattern of a given RNN via validated numerical dynamics techniques?This question is then divided into two sub-questions: what patterns are possible in a given RNN? and which of them is stable? The main ingredient towards this goal is the use of validated numerics outside of its traditional environment. The central theme of validated numerics is to turn numerical computations into proofs. This is usually achieved by considering a numerical approximation and rigorously constructing a posteriori bounds of its error, proving the existence of a solution in the same sweep. This project requires a background in network based dynamical systems and in validated numerics, both being in the applicant's core technical capabilities.

这个项目的目标是利用动力系统中经过验证的计算机辅助技术来理解递归神经网络(RNN)的行为。RNN是最近的机器学习算法中的一个关键组件，特别是在深度神经网络的背景下。RNN可以解释为图或网络上的动态系统。节点和边分别与状态值和权重相关联。首先，具有固定链路权重的神经网络的节点的信息处理是一个动态过程，因为来自输入层的初始条件被处理，并通过映射的有限时间迭代或有限时间流产生输出层的值。其次，神经网络的学习阶段也可以解释为通过迭代得到的动态最小化问题。当然，在网络上的动态之间，即在节点和链路上，存在反馈。在处理足够数量的训练数据得到的动态过程结束时，人们希望动态系统已经产生了足够稳定的链路权模式，该模式对于模式匹配甚至超越初始训练数据的外推等任务具有足够的表达能力，我们感兴趣的是研究稳定的网络结构，即权值将实现的模式。这种模式非常有趣，因为它们表明RNN何时接受了足够的培训，以及培训目标是否已经实现。这项研究还将用于比较不同起始条件的流和具有不同体系结构的RNN中的流。这将使我们能够为所考虑的应用程序选择RNN和系统架构的最佳组合。应用来自经过验证的数值计算的概念，将可能找到模式并证明它们的存在和稳定性。这将允许我们从数学上确定它们在给定的RNN中的出现情况。因此，本项目围绕以下问题展开：我们能否通过有效的数值动力学技术严格预测给定RNN的模式？然后，这个问题被分为两个子问题：在给定的RNN中，什么模式是可能的？他们中的哪一个是稳定的？实现这一目标的主要因素是在其传统环境之外使用经过验证的数字。验证数值的中心主题是将数值计算转化为证据。这通常是通过考虑数值逼近并严格构造其误差的后验界来实现的，从而在同一扫描中证明解的存在性。该项目要求有基于网络的动力系统和验证数值的背景，这两者都是申请者的核心技术能力。