CAREER: From Analysis to Practice: Landscape-driven Optimization Algorithms for Deep Learning

深度学习（DL）是科技行业的主要推动力，它被用于解决图像、语音和视频识别、图像分割和自然语言处理等大量问题。深度学习也越来越多地用于物理、医学和化学等学科。在任何这些应用中训练DL模型都需要解决一个数学问题，而这个数学问题的性质还不太清楚。因此，现有的深度学习训练方法是次优的，并且消耗大量的资源、时间和金钱。我们对深度学习的有限理解损害了所有依赖深度学习技术的公共和私营部门的进步，并限制了其在新应用中的部署。该项目旨在通过描述跨各种深度学习模型和数据集的深度学习系统的通用属性来克服这一限制。所获得的知识将被用于开发新一代的训练策略，这些策略是为DL环境量身定制的，并且是高效、准确和可扩展的。新的算法工具将对广泛的应用产生强大的影响，美国公共和私营实体可以利用这些工具，在需要处理大型复杂数据的人工智能业务的各个领域转向更强大的计算学习平台。该项目更广泛的影响活动包括(a)研究生和本科课程开发，(b)通过纽约大学科学与工程应用研究创新项目为高中生提供暑期研究机会，以及(c)通过研究者组织的纽约大学坦顿欧洲经委会现代人工智能系列研讨会普及知识，该研讨会向大学、高中和行业开放，并在全球范围内传播。本研究采用多层次的方法探索深度学习优化和泛化的原理，并开发新一代深度学习优化工具。首先，研究人员将寻求理解非凸深度学习损失景观的几何性质与深度学习模型的泛化能力之间的关系。接下来，研究人员将描述常见深度学习优化器的训练轨迹。这些研究对于开发具有景观意识的深度学习优化器至关重要。得到的优化器将被并行化，以便与通常用于在海量数据上训练大规模深度学习网络的计算机集群的体系结构兼容。新的并行优化器将在训练期间适应计算资源的动态分配，并且能够处理非常大的数据批。该奖项反映了美国国家科学基金会的法定使命，并通过使用基金会的知识价值和更广泛的影响审查标准进行评估，被认为值得支持。

Deep learning (DL) is a major driving force of tech industry, where it is used for a plethora of problems such as image, speech, and video recognition, image segmentation, and natural language processing. DL is also increasingly more often used in physics, medicine, and chemistry, among other disciplines. Training a DL model in any of these applications requires solving a mathematical problem whose properties are poorly understood. Consequently, existing DL training methodologies are sub-optimal and consume a large amount of resources, time, and money. Our limited understanding of DL compromises the progress of all public and private sectors that rely on DL technology, and limits its deployment in new applications. This project aims at overcoming this limitation by describing universal properties of DL systems that hold across a variety of DL models and data sets. The acquired knowledge will be used to develop a new generation of training strategies that are tailored to the DL setting and are efficient, accurate, and scalable. New algorithmic tools will have a strong impact on a wide range of applications and can be leveraged by US public and private entities to shift to significantly more powerful computational learning platforms in various areas of their AI-based businesses that require the processing of large and complex data. Broader impact activities of this project include (a) graduate and undergraduate curriculum development, (b) summer research opportunities for high-school students via NYU Applied Research Innovations in Science and Engineering program, and (c) knowledge popularization via NYU Tandon ECE Seminar Series on Modern AI, organized by the investigator, that is open to universities, high schools, and industry, and that is streamed worldwide.The proposed research is a multi-level approach to explore the principles of DL optimization and generalization, and to develop new generation DL optimization tools. First, the researchers will seek to understand the relationship between the geometric properties of the non-convex DL loss landscape and the generalization abilities of DL models. Next, the researchers will characterize the training trajectories of common DL optimizers. These studies will be essential for developing landscape-aware DL optimizers. The obtained optimizers will be parallelized in order to be compatible with the architecture of the computer clusters that are typically used to train large-scale DL networks on massive data. The new parallel optimizers will accommodate dynamic allocation of computational resources during training and will be able to process extremely large data batches.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.