CAREER: Coding Theory for Robust Large-Scale Machine Learning

编码理论通过支持信息在多方面不确定性背景下的鲁棒性，在现代信息技术中发挥了关键作用。随着最近在机器学习方面的成功，鲁棒性已经成为一种理想的原则，但现在是在大规模计算的背景下。在实际应用程序和非策划设置（通常是非理想环境）中部署机器学习解决方案时，与鲁棒性相关的挑战非常普遍。该项目旨在通过开发基于计算编码理论的新解决方案来解决这些挑战。这些解决方案提供了可证明的鲁棒性保证，在实践中可以胜过更传统的解决方案，并将已经改变了通信和存储系统的收益扩展到机器学习系统。研究人员现有的和新的合作将促进行业合作，并增加从该项目产生的框架和算法向实践的过渡。该研究将与教育科学最新进展指导下的教育发展紧密结合，并与威斯康星探索研究所的外展计划一起进行。该项目旨在为鲁棒的大规模机器学习开发新的编码理论解决方案和基本权衡。该研究项目围绕三个重点展开。第一个重点是在存在延迟和离散节点的分布式优化过程中的鲁棒性，其中收敛速度受到系统中明显慢于平均速度的节点的影响。第二个重点是在存在拜占庭节点和最坏情况故障的分布式优化过程中的鲁棒性。最近的研究提出了鲁棒聚合规则来过滤掉最坏情况或对抗性失败的影响。该项目开发的编码理论解决方案可以快几个数量级，并在计算和拜占庭容忍之间产生未探索的权衡。第三个重点是预测期间的对抗性扰动，它可以迫使最先进的模型始终错误地分类事件/数据。该项目的编码理论方法通过具有固有冗余的模型集成和数据增强来追求可证明的防御机制，以抵御对抗性攻击。所提出的理论和算法解决方案是由信息和编码理论、分布式优化和机器学习的跨学科工具组合提供的。该奖项反映了美国国家科学基金会的法定使命，并通过使用基金会的知识价值和更广泛的影响审查标准进行评估，被认为值得支持。

Coding theory has played a critical role in modern information technology by supporting robustness of information against a backdrop of multifaceted uncertainty. Following recent successes in machine learning, robustness has emerged as a desired principle, but now in the context of large-scale computation. Challenges related to robustness are prevalent when deploying machine learning solutions in real applications and non-curated settings, which are often non-ideal environments. This project aims to address these challenges by developing novel solutions based on coding theory for computation. These solutions offer provable robustness guarantees, can outperform more traditional solutions in practice, and extend to machine learning systems the gains that have transformed communication and storage systems. Existing and new collaborations of the investigator will facilitate industry cooperation and increase the transition to practice for the frameworks and algorithms generated from this project. The research will be strongly coupled with educational developments guided by recent advances in education science, alongside an outreach program within the Wisconsin Institute for Discovery. This project aims to develop novel coding-theoretic solutions and fundamental trade-offs for robust large-scale machine learning. The research program is centered around three thrusts. The first thrust focuses on robustness during distributed optimization in the presence of delays and straggler nodes, where the speed of convergence is affected by nodes in the system that are significantly slower than average. The second thrust focuses on robustness during distributed optimization in the presence of Byzantine nodes and worst-case failures. Recent studies proposed robust aggregation rules to filter out the effect of worst-case or adversarial failures. This project develops coding-theoretic solutions that can be orders of magnitude faster, and give rise to unexplored trade-offs between computation and Byzantine tolerance. The third thrust focuses on adversarial perturbations during prediction that can force state-of-the-art models to consistently mis-classify events/data. The coding-theoretic approach of this project pursues provable defense mechanisms against adversarial attacks through ensembles of models with inherent redundancy and through data augmentation. The proposed theoretical and algorithmic solutions are afforded by an interdisciplinary mix of tools from information and coding theory, distributed optimization, and machine learning.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.