Scaling Laws of Deep Learning

该项目通过描述应用科学家和工程师观察到的控制经验现象的基本量和一般规律，建立了深度学习的数学和科学基础。深度学习是机器学习和人工智能的一个范例，通过设计参数化模块网络并使用优化算法在大数据集上对其进行训练，从数据中学习统计模型。它对科学和社会有着广泛的影响，从自动驾驶汽车到在线商务和社交媒体。科学研究越来越依赖于深度学习来进行数据驱动的科学发现。这个项目汇集了一个由统计学家、数学家、计算机科学家和电气工程师组成的多学科团队。研究成果影响了许多核心学科和智能增强技术，为深度学习的实际应用提供了科学指导。该研究计划涉及缩放定律的概念。在实际应用方面，尺度定律极大地简化了大型实验的参数设置和模型向新领域的转移。在理论方面，标度定律揭示了经验现象，并将它们与数学的简洁统一起来。该团队以最优传输、经验过程、非参数统计、信息理论和复杂性理论的最新进展为基础，并将其工作建立在自然语言处理和计算机视觉等应用领域的大型实验的经验观察基础上。这些研究成果为利用深度学习解决具有挑战性问题的科学家和工程师提供了实用指南，也构成了数学、统计和计算机科学核心领域的基础进展。该奖项反映了美国国家科学基金会的法定使命，并通过使用基金会的知识价值和更广泛的影响审查标准进行评估，被认为值得支持。

This project builds the mathematical and scientific foundations of deep learning by characterizing the fundamental quantities and general laws that govern the empirical phenomena observed by applied scientists and engineers. Deep learning is a paradigm in machine learning and artificial intelligence where statistical models are learned from data by designing networks of parameterized modules and training them on big datasets using optimization algorithms. It has widespread effects on science and society, from autonomous vehicles to online commerce and social media. Scientific research increasingly relies on deep learning for data-driven scientific discovery. This project brings together a multidisciplinary team of statisticians, mathematicians, computer scientists, and electrical engineers. Research outcomes affect many core academic disciplines and intelligence augmentation technologies by providing scientific guidelines for practical applications of deep learning.The research program addresses the concept of scaling laws. On the practical side, scaling laws greatly simplify parameter setting for large experiments and model transfer to new domains. On the theoretical side, scaling laws shed light on empirical phenomena and unify them with mathematical concision. The team builds upon recent advances in optimal transport, empirical processes, nonparametric statistics, information theory, and complexity theory, and grounds its work in empirical observations made in large experiments in natural language processing and computer vision, among other applied domains. The research outcomes provide practical guidelines for scientists and engineers who employ deep learning to tackle challenging problems, and also constitute fundamental advances in the core areas of mathematical, statistical, and computer sciences.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.