CAREER: Variational Inference for Resource-Efficient Learning

由于大量的数据需求和参数丰富的模型，深度学习（DL）的强大带来了巨大的能量和存储成本。例如，最近用于自然语言生成的模型包含超过1000亿个参数，并且需要大量的训练数据。训练这样一个模型所需要的二氧化碳排放量几乎是普通美国汽车寿命的五倍。该项目开发了一种基于一套通用方法的资源高效深度学习的整体方法：通过信息论的视角来看待深度学习模型和算法，从而可以正式量化和最小化所需资源。该项目的成果包括压缩模型（神经网络）和数据（图像和视频）的新方法，以及用于深度学习的新训练算法，这些算法减少了数据需求并提高了运行效率。这些研究活动还将为代表性不足的学生提供夏季教学活动，为资源高效机器学习提供新的开源软件，以及神经压缩和统计机器学习的研讨会和专题讨论会。更详细地说，该项目从变分推理（VI）的角度接近资源高效机器学习，并包含三个重点，重点关注不同的低效率：(A)带宽效率低下：模型对数据或参数的低效表示，(B)数据效率低下：模型对训练数据的广泛需求，以及(C)运行时效率低下：学习或推理算法无法在给定的计算时间预算内产生所需的答案。Thrust A利用VI和率失真理论之间的联系，推导出新的具有改进压缩性能的神经压缩算法。推力B设计了信息先验，用于在非平稳环境中使用有限数据进行有效学习。最后，thrust C为混合了马尔可夫链蒙特卡罗和VI的贝叶斯神经网络开发了高度可扩展的训练算法，在收敛速度上权衡了精度。该项目包含图像和视频压缩以及气候科学领域的应用。该奖项反映了美国国家科学基金会的法定使命，并通过使用基金会的知识价值和更广泛的影响审查标准进行评估，被认为值得支持。

The power of Deep Learning (DL) comes with enormous energy and storage costs due to massive data needs and parameter-rich models. For example, recent models for natural language generation contain more than a hundred billion parameters and require huge amounts of training data. Training such a model can entail nearly five times the lifetime carbon dioxide emissions of the average American car. This project develops a holistic approach to resource-efficient DL based on a common set of methodologies: DL models and algorithms are viewed through the lens of information theory, making it possible to formally quantify and minimize the required resources. Outcomes of this project include new methods for the compression of both models (neural networks) and data (images and video), as well as new training algorithms for DL that reduce data requirements and improve runtime efficiency. These research activities will also inform summer teaching activities for under-represented students, lead to new open-source software for resource-efficient machine learning, as well as workshops and symposia on neural compression and statistical machine learning. In more detail, the project approaches resource-efficient machine learning from the perspective of variational inference (VI) and contains three thrusts that focus on different inefficiencies: (A) bandwidth inefficiency: a model's inefficient representation of data or parameters, (B) data inefficiency: a model's extensive need for training data, and (C) runtime inefficiency: a learning or inference algorithm's inability to produce desired answers within a given computational time budget. Thrust A draws on the connection between VI and rate-distortion theory to derive new neural compression algorithms with improved compression performance. Thrust B designs informative priors for effective learning with limited data in non-stationary environments. Finally, thrust C develops highly scalable training algorithms for Bayesian neural networks that hybridize Markov Chain Monte Carlo and VI, trading-off precision for convergence speed. The project contains applications from the domains of image and video compression as well as climate science.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.