CIF: Small: Deep Stochasticity for Private Collaborative Deep Learning

到目前为止，深度学习在从计算机视觉到自然语言处理再到药物设计的许多应用程序中都取得了前所未有的性能水平。培训模型通常需要大量的培训数据，这些数据是从多个个人/组织收集的，以确保异质性，因为同质数据可能会导致过度拟合。培训数据通常包含敏感信息，例如医疗记录、浏览历史或金融交易，从而对收集数据的个人构成隐私威胁。尽管分散学习和联合学习等多机器协作学习据称通过从不让原始训练数据离开参与机器来解决隐私问题，但最近的研究揭示了一幅完全不同的图景：不仅可以从共享的梯度/模型更新推断训练数据的特征，而且甚至可以从这些共享的梯度反向推断原始数据。此外，只有当噪声足够大时，将噪声添加到共享梯度才有效，这可能会导致训练精度的降低。相反，这个项目寻求通过明智的随机化来保护参与机器的隐私，这种随机化利用了协作学习算法的结构，并利用了它们对错误的天然弹性。该项目将通过为本科生和研究生班级提供关于保护隐私的分散学习的新模块来丰富当前的课程。将通过克莱姆森PEER(教育丰富和保留方案)和WISE(科学和工程女性)开展让少数族裔学生参与的外联活动，以扩大对计算机的参与。该项目探索了几种不同的方法，在算法层面明智地嵌入随机性，即所谓的深度随机性，以便在协作学习过程中实现隐私保护。所提出的方法利用了深度学习算法对参数错误/噪声的天然弹性，并且在不影响精度或引起大量计算/通信开销的情况下实现隐私，并具有适应诸如密码术等附加机制的灵活性。该方法既适用于无参数服务器的分布式学习，也适用于参数服务器促进的联合学习。主要研究集中在协作学习算法的设计上，该算法使用随机量化方案进行机器间通信，并在构建迭代时使用随机学习步长。将开发严格的分析框架来定量评估正在实现的隐私保护的力度，理论结果将通过机器人网络实验进行系统验证。该奖项反映了NSF的法定使命，并通过使用基金会的智力优势和更广泛的影响审查标准进行评估，被认为值得支持。

By now, Deep Learning is achieving unprecedented performance levels in many applications ranging from computer vision to natural language processing to drug design. Training models usually require large volumes of training data, said data being collected from multiple individuals/organizations to ensure heterogeneity since homogeneous data may lead to over-fitting. Training data often contain sensitive information, e.g., healthcare records, browsing history, or financial transactions, thereby posing privacy threats for the individuals from whom the data were collected. Although multi-machine collaborative learning, such as decentralized learning and federated learning, allegedly solves privacy concerns by never letting the raw training data leave the participating machines, recent studies have revealed a completely different picture: Not only can features of the training data be inferred from shared gradient/model updates, but even the raw data can be reversely inferred from these shared gradients. Moreover, adding noise to shared gradients, a de facto standard for achieving differential privacy, becomes effective only when the noise is sufficiently large, possibly leading to a degradation of the training accuracy. This project, instead, seeks to enable privacy protection for participating machines through judicious randomization that exploits the structure of collaborative learning algorithms and leverages their natural resiliency to error. The project will enrich the current curriculum by providing new modules on privacy-preserving decentralized learning for both undergraduate and graduate classes. Broadening Participation in Computing will be addressed through outreach activities involving minority students via Clemson PEER (Programs for Educational Enrichment and Retention) and WISE (Women in Science and Engineering). The project explores several different approaches to judiciously embed stochasticity at the algorithmic level, so-called deep stochasticity, in order to enable privacy protection in the collaborative learning process. The proposed approach exploits the natural resiliency of deep learning algorithms to parameter errors/noises, and enables privacy without compromising accuracy or incurring heavy computation/communication overheads with the flexibility to accommodate additional mechanisms like cryptography. The techniques are applicable to both parameter-server-free decentralized learning and to parameter-server facilitated federated learning. The main research thrusts center on the design of collaborative learning algorithms that use stochastic quantization schemes for inter-machine communications and random learning stepsizes in building the iterates. Rigorous analysis frameworks will be developed to quantitatively evaluate the strength of the privacy protection being achieved, and the theoretical results will be systematically validated through experiments with robot networks.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.