Dynamical systems approach to robust reconstruction of probability distributions of observed data

点击翻译按钮获取中文摘要

Our proof-of-concept project aims at constructing a bridge between machine learning and the theory of dynamical systems. More specifically, we develop a dynamical systems paradigm for learning probability distributions of observed data.Artificial neural networks have found a substantial number of applications in recent years due to their ability to represent probability distributions in a parametric way, and effectively learn their parameters from observed data. In our project, we study how the parameters of probability distributions (that are outputs of neural networks) evolve during learning. By considering the limit as the number of data points goes to infinity, we can describe the process of learning the parameters of distributions by a tractable system of differential equations, which we will analyze in detail. As we explain below, these dynamical systems typically possess families of equilibria that correspond to suboptimal values of the parameters, and as such the learning process may converge to an incorrect distribution. We emphasize that these suboptimal equilibria are not a result of over- or underparametrization by the weights of a neural network, but are inherent in the original parametrization of the approximating distributions. Furthermore, the structure and the stability of these suboptimal equilibria is affected by outliers in the training data.Our goal is to perform a detailed analysis of all the equilibria, their stability, basins of attraction, and the structure of their stable manifolds. We will use this knowledge to understand how one can modify the learning process in such a way that the parameters converge to the correct equilibrium and thus represent the correct ground truth distribution. In particular, our research programme will include the analysis of the dynamics corresponding to single- and multicomponent mixture distributions in the presence of outliers in the training data set. We especially aim at obtaining proper correction formulas for the variance in the presence of outliers in the training data set, which will be derived via a rigorous analysis of the attractors of the emerging dynamical systems.Although our methods heavily rely upon the theory of differential equations and dynamical systems, the anticipated results will be relevant to modern fields of stochastics, machine learning, and artificial intelligence. We believe that they will not only elucidate the pitfalls of learning probability distributions with neural networks, but also help to make the learning process more efficient.