Optimization and game dynamics in machine learning

本提案的重点是在ML中优化和游戏的数值方法背景下对动力学和加速度的研究。有大量关于优化的文献，特别是关于机器学习（ML）的优化。然而，仍有一些基本问题没有得到解答。令人惊讶的是，这些未解决的问题包括广泛使用的方法的基本特性，加速和随机性的相互作用，游戏的动力学，以及加速，适应性和泛化之间的联系。这种缺乏理解的后果是，我们不能充分受益于现有的方法，并进一步忽视了改进方法的可能性。我建议对优化和游戏动力学进行基础研究，因为它们与机器学习和深度学习中的现代挑战有关。这项工作包括对经典动量动力学的深入研究，强调一些以前未知和非常奇怪的性质；一种简单的加速随机优化方法博弈动力学中的加速度和随机性研究并着重从动力学角度研究了自适应优化方法的泛化能力。

This proposal focuses on the investigation of dynamics and acceleration in the context of numerical methods for optimization and games in ML. There is vast literature on optimization and specifically on optimization for machine learning (ML). However, there are a number of fundamental questions that are still unanswered. Those unanswered questions surprisingly include fundamental properties of widely used methods, the interaction of acceleration and stochasticity, the dynamics of games, and the connection between acceleration, adaptivity and generalization. A consequence of this lack of understanding is that we do not fully benefit from existing methods, and further ignore the possibility of improved methods. I am proposing fundamental research on optimization and game dynamics, as they pertain to modern challenges in machine learning and deep learning. The work includes a deeper study of classic momentum dynamics, emphasizing on some previously unknown and very curious properties; a simple approach to accelerated stochastic optimization; the study of acceleration and stochasticity on game dynamics; and an investigation of the generalization abilities of adaptive optimization methods with emphases on the lens of dynamics.