A Non-Asymptotic Analysis of Stochastic Mirror Descent for Non-Convex Learning

非凸学习的随机镜像下降的非渐近分析

• 批准号: 2444063
• 金额: --
• 依托单位: University of Oxford
• 依托单位国家: 英国
• 项目类别: Studentship
• 财政年份: 2020
• 资助国家: 英国
• 起止时间: 2020 至 无数据
• 项目状态: 未结题
• 来源: https://gtr.ukri.org/projects?ref=studentship-2444063
• 关键词: Non; Asymptotic; Analysis; Stochastic; Mirror

Many state of the art machine learning techniques depend heavily on the optimisation of some non-convex objective. Often, this procedure is computationally expensive and requires techniques that employ stochastic approximations or regularisation using artificial noise. For instance, there are a variety of techniques in deep learning that use artificial noise applied to the data, parameters or update direction, all of which have been shown to encourage faster convergence and improved generalisation. Though many of these techniques have demonstrated their validity through repeated experimentation and testing, the theoretical framework to validate and understand them is still in its infancy.Another technique which is used to overcome the aforementioned computational difficulties is the mirror descent framework, which does so by taking advantage of the known geometric properties specific to the problem in consideration. It is highly popular in convex optimisation as in this setting there are many provable advantages to this algorithm. Its stochastic counterpart, stochastic mirror descent, is rapidly gaining popularity in non-convex learning. But again, the theoretical framework in this setting is under-developed.Through our project, we seek to develop a better understanding of the statistical and computational properties of these techniques in the large-scale learning setting. Specifically, we are interested in quantitively comparing the consistency and generalisation performance of these techniques and seeing how they change as the problem grows more challenging.Guiding much of our methodology will be the focus on non-asymptotic analysis, that is compared to analyses in which quantities are taken to their infinite limits. We believe this is essential for applications in large-scale learning where only a limited number of iterations can take place and the effect of the number of parameters on performance is of key interest.Fundamentally, our approach will be based on approximating the iterative optimisation procedure with stochastic processes that evolve continuously in time, specifically diffusion processes. With this, we gain access to the broad suite of powerful techniques for analysing diffusions. This approach has seen a growing popularity in recent years and has already yielded interesting results in the learning setting. The project will also draw heavily from the study of stochastic differential equations as well as high dimensional statistics.This project falls within the EPSRC Statistics and Applied probability area.

许多最先进的机器学习技术在很大程度上依赖于一些非凸目标的优化。通常，该过程在计算上是昂贵的，并且需要采用随机近似或使用人工噪声的正则化的技术。例如，在深度学习中有各种技术使用应用于数据、参数或更新方向的人工噪声，所有这些技术都被证明可以促进更快的收敛和改进的泛化。虽然这些技术中的许多已经通过反复的实验和测试证明了它们的有效性，但是验证和理解它们的理论框架仍然处于起步阶段。另一种用于克服上述计算困难的技术是镜像下降框架，该框架通过利用特定于所考虑问题的已知几何特性来实现。它在凸优化中非常流行，因为在这种情况下，该算法有许多可证明的优点。它的随机对应物，随机镜像下降，在非凸学习中迅速流行。但同样，在这种情况下的理论框架是欠发达的。通过我们的项目，我们试图在大规模的学习环境中发展这些技术的统计和计算特性的更好的理解。具体来说，我们感兴趣的是定量比较这些技术的一致性和泛化性能，并看到它们如何随着问题变得更具挑战性而变化。指导我们的大部分方法将集中在非渐近分析上，即与数量被带到无限极限的分析进行比较。我们相信这是必不可少的应用程序在大规模的学习，只有有限数量的迭代可以发生和性能上的参数的数量的影响是关键interests.Fundamentally，我们的方法将是基于近似的迭代优化过程与随机过程，不断演变的时间，特别是扩散过程。有了这个，我们可以使用一系列强大的技术来分析扩散。近年来，这种方法越来越受欢迎，并且已经在学习环境中产生了有趣的结果。该项目还将大量借鉴随机微分方程以及高维统计的研究。该项目属于EPSRC统计和应用概率领域的福尔斯。