III: Small: Stochastic Algorithms for Large Scale Data Analysis

随机梯度下降(SGD)等随机算法是现代数据科学的主力。这样的算法在深度学习的成功中发挥了重要作用。尽管取得了这样的经验成功，但SGD在挑战深度学习中遇到的非凸优化问题时的行为仍然笼罩在神秘之中。关于SGD如何导航非凸损失景观，如何避免局部极小值，以及使用SGD学习的深度模型如何在未来数据上很好地推广，人们的理解有限。该项目的重点是澄清对SGD动力学的理解，并对深度学习背景下出现的非凸性问题进行泛化。该项目还使用改进的理解来开发分段方法，以自适应地使用验证集来选择超参数并避免过度拟合。从技术进步中获得的见解被应用于亚季节性到季节性(S2S)天气预报这一具有挑战性的科学问题，该问题侧重于预测几周至几个月的时间框架内的天气。S2S预报的进展对于水资源管理、农业、能源、航空、海事规划和应急计划等广泛的应用领域至关重要。该项目还吸引了更广泛的数据科学界，纳入了在丰富课程方面获得的见解，并扩大了未提交报告的群体的参与。该项目研究SGD动力学，主要关注过度参数设置，即样本数量小于参数数量，这是深度学习的典型情况。基于两个关键矩阵：非凸损失函数的Hessian矩阵和随机梯度的协方差矩阵、它们的本征谱以及它们的主子空间之间的重叠，对动力学进行了仔细的研究。虽然SGD动力学发生在高维空间中，但这些矩阵的主子空间可以是低维的。来自高维几何和相关随机过程的工具被用来描述高维空间中的这种低维动力学。基于这些矩阵的性质，还提出了解释用SGD训练的深度学习模型有趣的泛化行为的原则性方法。此外，开发了基于差异隐私的机制，以自适应地使用验证集来选择超参数并避免深度学习中的过度匹配。该奖项反映了NSF的法定使命，并通过使用基金会的智力优势和更广泛的影响审查标准进行评估，被认为值得支持。

Stochastic algorithms such as stochastic gradient descent (SGD) are the workhorse of modern data science. Such algorithms have been playing an important role in the success of deep learning. In spite of such empirical success, the behavior of SGD for challenging non-convex optimization problems as encountered in deep learning is shrouded in mystery. There is limited understanding of how SGD navigates non-convex loss landscapes, how bad local minima are avoided, and how deep models learned using SGD generalize well on future data. The project focuses on gaining clarity of understanding of SGD dynamics and generalization for non-convex problems arising in the context of deep learning. The project also uses the improved understanding to develop prinipled approches to adaptively use validation sets to choose hyper-parameters and avoid overfitting. The insights gained from the technical advances are applied to the challenging scientific problem of sub-seasonal to seasonal (S2S) weather forecasting, which focuses on forecasting weather on a few weeks to few months time-frame. Advances in S2S forecasting is critically important to a wide variety of application domains including water resource management, agriculture, energy, aviation, maritime planning, and emergency planning. The project also engages the broader data science community, incorporating the gained insights for curricular enrichment, and broadening participation from underepresented groups. The project studys SGD dynamics with primary focus on the over-parameterized setting, i.e., where the number of samples is smaller than the number of parameters, which is typical for deep learning. The dynamics is carefully studied based on two key matrices: the Hessian of the non-convex loss function and the covariance matrix of the stochastic gradients, their eigen-spectra, and the overlap between their principal subspaces. Although the SGD dynamics happen in a high-dimensional space, the principal subspaces of these matrices can be low-dimensional. Tools from high-dimensional geometry and associated stochastic processes are utilized to characterize such low dimensional dynamics in high-dimensional spaces. Principled approaches to explain the intriguing generalization behavior of deep learning models trained with SGD are also developed based on the properties of these matrices. Further, differential privacy based mechanisms are developed for adaptively using validation sets for choosing hyper-parameters and avoiding over-fitting in deep learning.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.