Optimization and game dynamics in machine learning

该提案的重点是在ML中优化和游戏的数值方法的背景下研究动力学和加速。有大量关于优化的文献，特别是关于机器学习（ML）的优化。然而，仍有一些基本问题没有得到解答。令人惊讶的是，这些未回答的问题包括广泛使用的方法的基本属性，加速和随机性的相互作用，游戏的动力学，以及加速，自适应和泛化之间的联系。这种缺乏理解的后果是，我们不能从现有方法中充分受益，并进一步忽视了改进方法的可能性。我建议对优化和游戏动态进行基础研究，因为它们涉及机器学习和深度学习的现代挑战。这项工作包括更深入地研究经典的动量动力学，强调一些以前未知的和非常好奇的属性;一个简单的方法来加速随机优化;研究游戏动力学的加速和随机性;和调查的泛化能力的自适应优化方法，重点是动态的透镜。

This proposal focuses on the investigation of dynamics and acceleration in the context of numerical methods for optimization and games in ML. There is vast literature on optimization and specifically on optimization for machine learning (ML). However, there are a number of fundamental questions that are still unanswered. Those unanswered questions surprisingly include fundamental properties of widely used methods, the interaction of acceleration and stochasticity, the dynamics of games, and the connection between acceleration, adaptivity and generalization. A consequence of this lack of understanding is that we do not fully benefit from existing methods, and further ignore the possibility of improved methods. I am proposing fundamental research on optimization and game dynamics, as they pertain to modern challenges in machine learning and deep learning. The work includes a deeper study of classic momentum dynamics, emphasizing on some previously unknown and very curious properties; a simple approach to accelerated stochastic optimization; the study of acceleration and stochasticity on game dynamics; and an investigation of the generalization abilities of adaptive optimization methods with emphases on the lens of dynamics.