III: Small: Stochastic Algorithms for Large Scale Data Analysis

随机算法，如随机梯度下降（SGD）是现代数据科学的主力。这些算法在深度学习的成功中发挥了重要作用。尽管取得了这样的经验性成功，但SGD在深度学习中遇到的挑战性非凸优化问题的行为仍然笼罩在神秘之中。对于SGD如何导航非凸损失景观，如何避免糟糕的局部最小值，以及使用SGD学习的深度模型如何很好地推广未来数据的理解有限。该项目的重点是清晰地理解SGD动态和深度学习背景下出现的非凸问题的泛化。该项目还使用改进的理解来开发自适应地使用验证集来选择超参数并避免过拟合的方法。从技术进步中获得的见解适用于具有挑战性的科学问题，即季节性天气预报（S2S），其重点是预测几周到几个月的时间范围内的天气。S2S预测的进展对于水资源管理、农业、能源、航空、海事规划和应急规划等广泛的应用领域至关重要。该项目还吸引了更广泛的数据科学社区，将所获得的见解纳入课程丰富，并扩大了代表性不足的群体的参与。该项目研究SGD动态，主要关注过度参数化设置，即，其中样本的数量小于参数的数量，这对于深度学习是典型的。基于两个关键矩阵仔细研究了动态：非凸损失函数的Hessian和随机梯度的协方差矩阵，它们的特征谱，以及它们的主子空间之间的重叠。虽然SGD动力学发生在高维空间中，但这些矩阵的主子空间可以是低维的。工具从高维几何和相关的随机过程被用来表征这种低维动力学在高维空间。基于这些矩阵的性质，还开发了解释使用SGD训练的深度学习模型的有趣泛化行为的原则方法。此外，还开发了基于差异隐私的机制，用于自适应地使用验证集来选择超参数，并避免深度学习中的过度拟合。该奖项反映了NSF的法定使命，并通过使用基金会的知识价值和更广泛的影响审查标准进行评估，被认为值得支持。

Stochastic algorithms such as stochastic gradient descent (SGD) are the workhorse of modern data science. Such algorithms have been playing an important role in the success of deep learning. In spite of such empirical success, the behavior of SGD for challenging non-convex optimization problems as encountered in deep learning is shrouded in mystery. There is limited understanding of how SGD navigates non-convex loss landscapes, how bad local minima are avoided, and how deep models learned using SGD generalize well on future data. The project focuses on gaining clarity of understanding of SGD dynamics and generalization for non-convex problems arising in the context of deep learning. The project also uses the improved understanding to develop prinipled approches to adaptively use validation sets to choose hyper-parameters and avoid overfitting. The insights gained from the technical advances are applied to the challenging scientific problem of sub-seasonal to seasonal (S2S) weather forecasting, which focuses on forecasting weather on a few weeks to few months time-frame. Advances in S2S forecasting is critically important to a wide variety of application domains including water resource management, agriculture, energy, aviation, maritime planning, and emergency planning. The project also engages the broader data science community, incorporating the gained insights for curricular enrichment, and broadening participation from underepresented groups. The project studys SGD dynamics with primary focus on the over-parameterized setting, i.e., where the number of samples is smaller than the number of parameters, which is typical for deep learning. The dynamics is carefully studied based on two key matrices: the Hessian of the non-convex loss function and the covariance matrix of the stochastic gradients, their eigen-spectra, and the overlap between their principal subspaces. Although the SGD dynamics happen in a high-dimensional space, the principal subspaces of these matrices can be low-dimensional. Tools from high-dimensional geometry and associated stochastic processes are utilized to characterize such low dimensional dynamics in high-dimensional spaces. Principled approaches to explain the intriguing generalization behavior of deep learning models trained with SGD are also developed based on the properties of these matrices. Further, differential privacy based mechanisms are developed for adaptively using validation sets for choosing hyper-parameters and avoiding over-fitting in deep learning.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.