Validated computation of patterns in recurrent neural networks

该项目的目标是采用来自动力系统的经过验证的计算机辅助技术来理解递归神经网络（RNN）的行为。RNN是最近机器学习算法的一个关键组成部分，特别是在深度神经网络的背景下。RNN可以被解释为图或网络上的动态系统。节点和边分别与状态值和权重相关联。首先，具有固定链接权重的神经网络节点的信息处理是一个动态过程，因为来自输入层的初始条件被处理，并且通过映射的有限时间迭代或有限时间流产生输出层的值。其次，神经网络的学习阶段也可以被解释为通过迭代获得的动态最小化问题。当然，在网络上和网络的动态之间存在反馈，即，在节点和链路上。在通过处理足够数量的训练数据而获得的动态过程结束时，人们希望动态系统已经产生了足够稳定的链接权重模式，该模式对于诸如模式匹配甚至超出初始训练数据的外推等任务具有足够的表达能力。这种模式非常有趣，因为它们表明RNN何时接受了足够的训练，以及训练目标是否已经实现。这项研究还将用于比较具有不同起始条件的流和具有不同架构的RNN中的流。这将使我们能够为所考虑的应用选择RNN和系统架构的最佳组合。应用验证数字的概念，可以找到模式并证明它们的存在性和稳定性。这将允许我们在数学上确定它们在给定RNN中的出现。因此，该项目围绕以下问题展开：我们能否通过有效的数值动力学技术严格预测给定RNN的模式？然后，这个问题被分成两个子问题：在给定的RNN中，什么模式是可能的？哪一个是稳定的实现这一目标的主要因素是在传统环境之外使用经过验证的数字。验证数值的中心主题是将数值计算转化为证明。这通常是通过考虑一个数值近似和严格构造其误差的后验界，证明在同一扫描的解决方案的存在性。这个项目需要在网络为基础的动力系统和验证数字的背景，都在申请人的核心技术能力。

The goal of this project is to employ validated computer-assisted techniques from dynamical systems to understand the behaviour of recurrent neural networks (RNNs). RNNs are one key component in recent machine learning algorithms, particularly in the context of deep neural networks. RNNs can be interpreted as dynamical systems on graphs, or networks. Nodes and edges are associated to state values and weights respectively. First, information processing by the nodes of a neural net with fixed link weights is a dynamical process as initial conditions from the input layer are processed and yield, via a finite-time iteration of maps or a finite-time flow, the values at the output layer. Second, also the learning phase of a neural network can be interpreted as a dynamical minimization problem obtained by iteration. Of course, there is a feedback between the dynamics on and of the network, i.e., on the nodes and of the links. At the end of the dynamical process obtained by processing a sufficient amount of training data, one hopes that the dynamical system has produced a sufficiently stable pattern of the link weights, which has sufficient expressive power for tasks such as pattern matching or even extrapolation beyond initial training data.We are interested in studying the stable network configuration, that is the pattern, that the weights will achieve. Such patterns are of great interest because they indicate when the RNN received sufficient training, and if the training objective has been achieved. This study would also be used to compare flows with different starting conditions and flows in RNNs with different architectures. This would allow us to choose the best combination of RNN and system architecture for the application under consideration.Applying concepts from validated numerics, it will be possible to find patterns and prove their existence and stability. This is going to allow us to mathematically determine their occurrence in a given RNN. Hence, this project centers around the following question:Can we rigorously predict the pattern of a given RNN via validated numerical dynamics techniques?This question is then divided into two sub-questions: what patterns are possible in a given RNN? and which of them is stable? The main ingredient towards this goal is the use of validated numerics outside of its traditional environment. The central theme of validated numerics is to turn numerical computations into proofs. This is usually achieved by considering a numerical approximation and rigorously constructing a posteriori bounds of its error, proving the existence of a solution in the same sweep. This project requires a background in network based dynamical systems and in validated numerics, both being in the applicant's core technical capabilities.