CIF: Small: Compression for Learning over networks

数据压缩是所有通信协议的核心组成部分，因为它可以转化为带宽节省、能源效率和低延迟操作。在传统的设置中，信息源对其消息进行压缩，以便能够有效地进行通信，以确保在目的地进行准确的重建。该项目旨在设计专门针对机器学习应用的压缩方案：如果传输的消息支持给定的学习任务(例如，分类或学习)，所需的压缩方案应该为学习任务提供更好的支持，而不是关注重建的准确性。这种压缩方法可能会在通信效率方面产生显著的好处，同时促进机器学习算法的成功实施。通过提高通信效率，这类方案有望有助于分布式机器学习算法在网络上的成功实现。传统上，压缩方案是使用率失真折衷来评估的；本项目感兴趣的是率精度折衷，其中精度捕捉量化可能对特定机器学习任务产生的影响。对于以下两个问题，人们特别感兴趣的是信息论的下界和权衡，以及对以下两个问题的显式压缩：(1)当我们需要使用分布式通信受限节点来快速有效地学习模型时，如何压缩模型训练；以及(2)如何在推理过程中压缩通信。该项目将为来自复合分布的特征导出分布式压缩的界限和算法，这些特征将用于机器学习任务，如分类。这项工作将推进最先进的技术，并在数据压缩和分布式机器学习领域建立新的联系。该奖项反映了NSF的法定使命，并通过使用基金会的智力优势和更广泛的影响审查标准进行评估，被认为值得支持。

Data compression is a core component of all communication protocols, as it can translate to bandwidth savings, energy efficiency and low delay operations. In the traditional setup, an information source compresses its messages so that they can be communicated efficiently with the goal of ensuring accurate reconstruction at the destination. This project seeks to design compression schemes that are specifically tailored to Machine Learning applications: If the transmitted messages support a given learning task (e.g., classification or learning), the desired compression schemes should provide better support for the learning task instead of focusing on reconstruction accuracy. This approach to compression could potentially yield significant benefits in terms of communication efficiency, while simultaneously promoting the successful implementation of Machine Learning algorithms. By improving communication efficiency, such schemes are expected to contribute to the successful implementation of distributed machine learning algorithms over networks.Traditionally, compression schemes are evaluated using rate-distortion trade-offs; this project is interested in rate-accuracy trade-offs, where accuracy captures the effect that quantization may have on a specific machine learning task. There is particular interest in information-theoretic lower bounds and trade-offs, and in explicit compression for the following two questions: (1) How to compress for model training, when we need to use distributed communication constrained nodes to learn a model, fast and efficiently; and (2) How to compress for communication during inference. The project will derive bounds and algorithms for distributed compression of features coming from composite distributions that will be used for a machine learning task, such as classification. This work will advance the state of the art, and build new connections between the areas of data compression and distributed machine learning.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.