CAREER: Variational Inference for Resource-Efficient Learning

由于海量数据需求和参数丰富的模型，深度学习(DL)的强大功能伴随着巨大的能源和存储成本。例如，最近的自然语言生成模型包含超过1000亿个参数，需要大量的训练数据。训练一辆这样的车型需要的二氧化碳排放量几乎是普通美国汽车使用寿命的五倍。该项目基于一套通用的方法开发了一种资源高效的数字图书馆的整体方法：数字图书馆的模型和算法是从信息论的角度来看待的，这使得正式地量化和最小化所需的资源成为可能。该项目的成果包括模型(神经网络)和数据(图像和视频)压缩的新方法，以及减少数据需求和提高运行效率的新的DL训练算法。这些研究活动还将为低于限额学生的暑期教学活动提供信息，产生新的开放源码软件，用于资源节约型机器学习，以及关于神经压缩和统计机器学习的讲习班和专题讨论会。更详细地说，该项目从变分推理(VI)的角度来处理资源高效的机器学习，并包含三个重点针对不同低效的推力：(A)带宽低效：模型对数据或参数的低效表示；(B)数据低效：模型对训练数据的广泛需求；以及(C)运行时低效：学习或推理算法无法在给定的计算时间预算内产生期望的答案。推力A利用VI和率失真理论之间的联系，推导出具有更好压缩性能的新的神经压缩算法。推力B为在非平稳环境中使用有限数据进行有效学习设计了信息先验。最后，推力C为贝叶斯神经网络开发了高度可扩展的训练算法，该算法结合了马尔可夫链蒙特卡罗和VI，以精度为代价换取了收敛速度。该项目包含图像和视频压缩以及气候科学领域的应用。该奖项反映了NSF的法定使命，并通过使用基金会的智力优势和更广泛的影响审查标准进行评估，被认为值得支持。

The power of Deep Learning (DL) comes with enormous energy and storage costs due to massive data needs and parameter-rich models. For example, recent models for natural language generation contain more than a hundred billion parameters and require huge amounts of training data. Training such a model can entail nearly five times the lifetime carbon dioxide emissions of the average American car. This project develops a holistic approach to resource-efficient DL based on a common set of methodologies: DL models and algorithms are viewed through the lens of information theory, making it possible to formally quantify and minimize the required resources. Outcomes of this project include new methods for the compression of both models (neural networks) and data (images and video), as well as new training algorithms for DL that reduce data requirements and improve runtime efficiency. These research activities will also inform summer teaching activities for under-represented students, lead to new open-source software for resource-efficient machine learning, as well as workshops and symposia on neural compression and statistical machine learning. In more detail, the project approaches resource-efficient machine learning from the perspective of variational inference (VI) and contains three thrusts that focus on different inefficiencies: (A) bandwidth inefficiency: a model's inefficient representation of data or parameters, (B) data inefficiency: a model's extensive need for training data, and (C) runtime inefficiency: a learning or inference algorithm's inability to produce desired answers within a given computational time budget. Thrust A draws on the connection between VI and rate-distortion theory to derive new neural compression algorithms with improved compression performance. Thrust B designs informative priors for effective learning with limited data in non-stationary environments. Finally, thrust C develops highly scalable training algorithms for Bayesian neural networks that hybridize Markov Chain Monte Carlo and VI, trading-off precision for convergence speed. The project contains applications from the domains of image and video compression as well as climate science.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.