AF: Small: Collaborative Research: Rigorous Approaches for Scalable Privacy-preserving Deep Learning

这个时代最显著的特征之一是大量个人和敏感数据的传播。尽管具有巨大的社会效益，但现代机器学习的强大工具，尤其是深度学习，可能对个人隐私构成真正的威胁。例如，在过去的几年里，深度神经网络在从大型复杂数据集中学习甚至是最精细的细节方面具有非凡的能力，这一点已经变得很明显。有了这些强大的工具，对隐私保护的有力和严格保证变得更加重要。过去十年见证了一种被称为差分隐私的可靠数学理论的兴起，它使设计数据分析算法能够对其输入数据集进行严格的隐私保证。尽管这一理论取得了显著的成功，但在处理复杂模型（如深度学习中出现的模型）时，来自差异隐私的现有工具在提供可接受的效用保证方面受到严重限制。该项目将通过提供新的原则方法来设计可扩展到工业工作负载的差异化私有深度学习算法，从而解决这些限制。该项目还将涉及与工业界的合作，这将有助于在实际数据集上评估已开发的算法，并开发开源软件工具。该项目的产品有可能改变现代机器学习系统中使用大量敏感数据的方式，这将影响这些系统在实践中的设计和实施方式。这个项目的活动还旨在通过征聘妇女和代表性不足群体的成员，促进计算机领域的多样性。研究人员将为现代机器学习开发一种严格的、多方面的设计范式，用于可扩展的、实用的、差异化的私有算法。该范式基于两种一般策略：(i)利用数据和机器学习模型的现实和有用属性来规避差分隐私文献中的现有限制，以及（ii）利用有限数量的公共数据（没有隐私约束）来提高算法的效用。基于这些策略，该项目将追求以下方向：(1)开发一个新的通用框架，用于在保护隐私的机器学习中利用公共数据；(2)设计改进的迭代训练算法，可以绕过所谓的差分隐私“组合定理”的标准使用；(3)设计新的差分私有随机梯度方法，专门针对非凸和过度参数化的机器学习问题。该奖项反映了美国国家科学基金会的法定使命，并通过使用基金会的知识价值和更广泛的影响审查标准进行评估，被认为值得支持。

One of the most salient features of this time is the dissemination of massive amounts of personal and sensitive data. Despite their enormous societal benefits, the powerful tools of modern machine learning, especially deep learning, can pose real threats to personal privacy. For example, over the last few years, it has become evident that deep neural networks have a remarkable power in learning even the finest details from large complex data sets. With such powerful tools, the need for robust and rigorous guarantees for privacy protection has become more crucial. The last decade has witnessed the rise of a sound mathematical theory, known as differential privacy, that enables designing data-analysis algorithms with rigorous privacy guarantees for their input data sets. Despite the noticeable success of this theory, existing tools from differential privacy are severely limited in offering acceptable utility guarantees when dealing with complex models like those arising in deep learning. This project will address those limitations by offering new principled approaches for designing differentially-private deep-learning algorithms that can scale to industrial workloads. The project will also involve collaboration with industry, which will facilitate the evaluation of the developed algorithms on real-world datasets and the development of open-source software tools. The products of this project have the potential to transform the way massive sets of sensitive data are used in modern machine-learning systems, which will impact the way these systems are designed and implemented in practice. The activities of this project will also aim at promoting diversity in computing by recruiting women and members of underrepresented groups.The investigators will develop a rigorous, multi-faceted design paradigm for scalable, practical, differentially private algorithms for modern machine learning. This paradigm is based on two general strategies: (i) exploiting realistic and useful properties of the data and the machine-learning models to circumvent existing limitations in the literature on differential privacy, and (ii) leveraging a limited amount of public data (that has no privacy constraints) to boost the utility of the algorithms. Based on these strategies, the project will pursue following directions: (1) developing a new, generic framework for utilizing public data in privacy-preserving machine learning, (2) designing improved iterative training algorithms that can bypass the standard use of the so-called "composition theorem" of differential privacy, and (3) designing new differentially private stochastic-gradient methods tuned specifically to non-convex and over-parameterized machine-learning problems.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.