CIF: Small: Deep Stochasticity for Private Collaborative Deep Learning

CIF：小：私人协作深度学习的深度随机性

• 批准号: 2215088
• 负责人: Yongqiang Wang
• 金额: $ 35万
• 依托单位: Clemson University
• 依托单位国家: 美国
• 项目类别: Standard Grant
• 财政年份: 2022
• 资助国家: 美国
• 起止时间: 2022-10-01 至 2025-03-31
• 项目状态: 未结题
• 来源: https://www.nsf.gov/awardsearch/showAward?AWD_ID=2215088&HistoricalAwards=false
• 关键词: CIF; Small; Deep; Stochasticity; Private

By now, Deep Learning is achieving unprecedented performance levels in many applications ranging from computer vision to natural language processing to drug design. Training models usually require large volumes of training data, said data being collected from multiple individuals/organizations to ensure heterogeneity since homogeneous data may lead to over-fitting. Training data often contain sensitive information, e.g., healthcare records, browsing history, or financial transactions, thereby posing privacy threats for the individuals from whom the data were collected. Although multi-machine collaborative learning, such as decentralized learning and federated learning, allegedly solves privacy concerns by never letting the raw training data leave the participating machines, recent studies have revealed a completely different picture: Not only can features of the training data be inferred from shared gradient/model updates, but even the raw data can be reversely inferred from these shared gradients. Moreover, adding noise to shared gradients, a de facto standard for achieving differential privacy, becomes effective only when the noise is sufficiently large, possibly leading to a degradation of the training accuracy. This project, instead, seeks to enable privacy protection for participating machines through judicious randomization that exploits the structure of collaborative learning algorithms and leverages their natural resiliency to error. The project will enrich the current curriculum by providing new modules on privacy-preserving decentralized learning for both undergraduate and graduate classes. Broadening Participation in Computing will be addressed through outreach activities involving minority students via Clemson PEER (Programs for Educational Enrichment and Retention) and WISE (Women in Science and Engineering). The project explores several different approaches to judiciously embed stochasticity at the algorithmic level, so-called deep stochasticity, in order to enable privacy protection in the collaborative learning process. The proposed approach exploits the natural resiliency of deep learning algorithms to parameter errors/noises, and enables privacy without compromising accuracy or incurring heavy computation/communication overheads with the flexibility to accommodate additional mechanisms like cryptography. The techniques are applicable to both parameter-server-free decentralized learning and to parameter-server facilitated federated learning. The main research thrusts center on the design of collaborative learning algorithms that use stochastic quantization schemes for inter-machine communications and random learning stepsizes in building the iterates. Rigorous analysis frameworks will be developed to quantitatively evaluate the strength of the privacy protection being achieved, and the theoretical results will be systematically validated through experiments with robot networks.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.

到目前为止，深度学习在从计算机视觉到自然语言处理再到药物设计的许多应用中达到了前所未有的性能水平。训练模型通常需要大量的训练数据，所述数据从多个个人/组织收集以确保异质性，因为同质数据可能导致过度拟合。训练数据通常包含敏感信息，例如，医疗保健记录、浏览历史或金融交易，从而对收集数据的个人构成隐私威胁。尽管多机器协作学习，如分散学习和联邦学习，据称通过从不让原始训练数据离开参与机器来解决隐私问题，但最近的研究揭示了一个完全不同的画面：不仅可以从共享梯度/模型更新中推断训练数据的特征，甚至可以从这些共享梯度中推断原始数据。此外，将噪声添加到共享梯度（用于实现差分隐私的事实上的标准）仅在噪声足够大时才变得有效，这可能导致训练精度的降低。相反，该项目旨在通过明智的随机化来保护参与机器的隐私，这种随机化利用了协作学习算法的结构，并利用了它们对错误的自然弹性。该项目将丰富现有课程，为本科生和研究生班提供关于隐私保护分散学习的新模块。将通过克莱姆森（教育充实和保留方案）和WISE（科学和工程领域的妇女）涉及少数民族学生的外联活动，扩大对计算的参与。该项目探索了几种不同的方法，以明智地在算法层面嵌入随机性，即所谓的深度随机性，以便在协作学习过程中实现隐私保护。所提出的方法利用了深度学习算法对参数错误/噪声的自然弹性，并在不影响准确性或产生大量计算/通信开销的情况下实现隐私，同时灵活地适应加密等附加机制。这些技术既适用于无参数服务器的分散学习，也适用于参数服务器促进的联邦学习。主要的研究重点集中在协作学习算法的设计上，该算法使用随机量化方案进行机器间通信，并在构建迭代时使用随机学习步长。将开发严格的分析框架，以定量评估所实现的隐私保护的强度，并通过机器人网络的实验系统地验证理论结果。该奖项反映了NSF的法定使命，并通过使用基金会的知识价值和更广泛的影响审查标准进行评估，被认为值得支持。

项目成果

Distributed Optimization with Noisy Information Sharing

• DOI: 10.23919/ccc58697.2023.10240327
• 发表时间: 2023-07
• 期刊: 2023 42nd Chinese Control Conference (CCC)
• 作者: Yongqiang Wang

Differentially-Private Distributed Optimization with Guaranteed Optimality

• DOI: 10.1109/cdc49753.2023.10383285
• 发表时间: 2023-12
• 期刊: 2023 62nd IEEE Conference on Decision and Control (CDC)
• 作者: Yongqiang Wang;A. Nedić

Quantization Enabled Privacy Protection in Decentralized Stochastic Optimization

• DOI: 10.1109/tac.2022.3198030
• 发表时间: 2022-08
• 期刊: IEEE Transactions on Automatic Control
• 影响因子: 6.8
• 作者: Yongqiang Wang;T. Başar

A Robust Dynamic Average Consensus Algorithm that Ensures both Differential Privacy and Accurate Convergence

• DOI: 10.1109/cdc49753.2023.10383541
• 发表时间: 2022-11
• 期刊: 2023 62nd IEEE Conference on Decision and Control (CDC)
• 作者: Yongqiang Wang

Differentially-private Distributed Algorithms for Aggregative Games with Guaranteed Convergence

保证收敛的聚合博弈的差分私有分布式算法

• DOI: 10.1109/tac.2024.3351068
• 发表时间: 2024
• 期刊: IEEE Transactions on Automatic Control
• 影响因子: 6.8
• 作者: Wang, Yongqiang;Nedić, Angelia
• 通讯作者: Nedić, Angelia