Optimization and game dynamics in machine learning

点击翻译按钮获取中文摘要

This proposal focuses on the investigation of dynamics and acceleration in the context of numerical methods for optimization and games in ML. There is vast literature on optimization and specifically on optimization for machine learning (ML). However, there are a number of fundamental questions that are still unanswered. Those unanswered questions surprisingly include fundamental properties of widely used methods, the interaction of acceleration and stochasticity, the dynamics of games, and the connection between acceleration, adaptivity and generalization. A consequence of this lack of understanding is that we do not fully benefit from existing methods, and further ignore the possibility of improved methods. I am proposing fundamental research on optimization and game dynamics, as they pertain to modern challenges in machine learning and deep learning. The work includes a deeper study of classic momentum dynamics, emphasizing on some previously unknown and very curious properties; a simple approach to accelerated stochastic optimization; the study of acceleration and stochasticity on game dynamics; and an investigation of the generalization abilities of adaptive optimization methods with emphases on the lens of dynamics.