AF: Small: Collaborative Research: Rigorous Approaches for Scalable Privacy-preserving Deep Learning

这一时期最显著的特征之一是大量个人和敏感数据的传播。尽管有巨大的社会效益，但现代机器学习的强大工具，特别是深度学习，可能会对个人隐私构成真正的威胁。例如，在过去的几年里，很明显，深度神经网络在从大型复杂数据集中学习即使是最精细的细节方面也具有非凡的能力。有了这样强大的工具，对隐私保护的强有力和严格保障的需要变得更加关键。过去十年见证了一种名为差分隐私的可靠数学理论的兴起，该理论使设计数据分析算法能够对其输入数据集提供严格的隐私保证。尽管这一理论取得了显著的成功，但在处理像深度学习中出现的复杂模型时，差异隐私的现有工具在提供可接受的效用保证方面受到严重限制。该项目将通过提供新的原则性方法来解决这些限制，以设计可扩展到工业工作负载的不同私有深度学习算法。该项目还将涉及与工业界的合作，这将促进在真实世界数据集上评价所开发的算法和开发开放源码软件工具。该项目的产品有可能改变现代机器学习系统中使用海量敏感数据集的方式，这将影响这些系统的设计和实施方式。该项目的活动还将旨在通过招募女性和代表性不足群体的成员来促进计算的多样性。研究人员将为现代机器学习的可伸缩、实用、不同私有的算法开发一种严格的、多方面的设计范例。该范例基于两个一般策略：(I)利用数据和机器学习模型的真实和有用的属性来绕过文献中关于差异隐私的现有限制，以及(Ii)利用有限数量的公共数据(没有隐私限制)来提高算法的实用性。基于这些策略，该项目将寻求以下方向：(1)开发一个新的通用框架，用于在隐私保护机器学习中利用公共数据，(2)设计改进的迭代训练算法，可以绕过所谓的差分隐私“合成定理”的标准使用，以及(3)设计专门针对非凸和过度参数的机器学习问题的新的差分私有随机梯度方法。该奖项反映了NSF的法定使命，并通过使用基金会的智力优势和更广泛的影响审查标准进行评估，被认为值得支持。

One of the most salient features of this time is the dissemination of massive amounts of personal and sensitive data. Despite their enormous societal benefits, the powerful tools of modern machine learning, especially deep learning, can pose real threats to personal privacy. For example, over the last few years, it has become evident that deep neural networks have a remarkable power in learning even the finest details from large complex data sets. With such powerful tools, the need for robust and rigorous guarantees for privacy protection has become more crucial. The last decade has witnessed the rise of a sound mathematical theory, known as differential privacy, that enables designing data-analysis algorithms with rigorous privacy guarantees for their input data sets. Despite the noticeable success of this theory, existing tools from differential privacy are severely limited in offering acceptable utility guarantees when dealing with complex models like those arising in deep learning. This project will address those limitations by offering new principled approaches for designing differentially-private deep-learning algorithms that can scale to industrial workloads. The project will also involve collaboration with industry, which will facilitate the evaluation of the developed algorithms on real-world datasets and the development of open-source software tools. The products of this project have the potential to transform the way massive sets of sensitive data are used in modern machine-learning systems, which will impact the way these systems are designed and implemented in practice. The activities of this project will also aim at promoting diversity in computing by recruiting women and members of underrepresented groups.The investigators will develop a rigorous, multi-faceted design paradigm for scalable, practical, differentially private algorithms for modern machine learning. This paradigm is based on two general strategies: (i) exploiting realistic and useful properties of the data and the machine-learning models to circumvent existing limitations in the literature on differential privacy, and (ii) leveraging a limited amount of public data (that has no privacy constraints) to boost the utility of the algorithms. Based on these strategies, the project will pursue following directions: (1) developing a new, generic framework for utilizing public data in privacy-preserving machine learning, (2) designing improved iterative training algorithms that can bypass the standard use of the so-called "composition theorem" of differential privacy, and (3) designing new differentially private stochastic-gradient methods tuned specifically to non-convex and over-parameterized machine-learning problems.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.