SaTC: CORE: Small: Detecting and Mitigating Unintended Learning

机器学习正在推动生物医学研究、自然语言处理、图像识别、自动驾驶汽车等领域的重大进步，这些进步取决于数据的持续可用性。 通过确保数据和基于这些数据的机器学习模型的完整性和隐私性，该项目旨在为所有数据持有者带来机器学习的好处。然而，机器学习可能会无意中泄露敏感数据，例如特定人员的身份。本项目将研究非预期学习，即，机器学习模型在其既定目标之外发现了什么，以及它对模型训练和应用的数据的影响。 在这个项目中，第一个重点领域是开发推理技术，检测敏感数据的泄漏，更一般地说，确定哪些模型实际上正在学习。这项研究将有助于确定深度模型中训练数据记忆的根本原因，开发检测和测量它的方法，并有助于防止深度模型无意中学习数据的隐私敏感特征。 第二个重点领域是开发和分析方法，以减少非预期的学习，并确保模型不包含不需要的或恶意的功能。 除了提高训练和测试数据的隐私性外，这项研究还将有助于检测和防止在智能手机和其他边缘设备上训练的模型中的后门。作为该项目一部分开发的所有技术都将在最先进的图像分析和自然语言模型上进行评估。该奖项反映了NSF的法定使命，并被认为值得通过使用基金会的知识价值和更广泛的影响审查标准进行评估。

Machine learning is fueling major advances in biomedical research, natural language processing, image recognition, self-driving vehicles, etc. These advances depend on the continuing availability of data. By assuring the integrity and privacy of both the data and the machine learning models based on this data, this project aims to bring the benefits of machine learning to all data holders. However, machine learning can unintentionally reveal sensitive data such as the identity of specific persons. This project will investigate unintended learning, i.e., what machine learning models are discovering beyond their stated objectives, and its consequences for the data on which the models are trained and to which they are applied. In this project, the first focus area is developing inference techniques that detect leakage of sensitive data and, more generally, determine what models are actually learning. This research will help identify the root causes of training-data memorization in deep models, develop methodology for detecting and measuring it, and help prevent deep models from unintentionally learning privacy-sensitive features of the data. The second focus area is developing and analyzing methods for mitigating unintended learning and ensuring that models do not contain unwanted or malicious functionality. In addition to improving privacy of the training and test data, this research will help detect and prevent backdoors in models trained on smartphones and other edge devices. All technologies developed as part of this project will be evaluated on state-of-the-art image-analysis and natural-language models.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.