CAREER: Enabling data valuation and deletion in human-centered machine learning

职业：在以人为本的机器学习中实现数据评估和删除

• 批准号: 1942926
• 负责人: James Zou
• 金额: $ 49.99万
• 依托单位: Stanford University
• 依托单位国家: 美国
• 项目类别: Continuing Grant
• 财政年份: 2020
• 资助国家: 美国
• 起止时间: 2020-06-15 至 2025-05-31
• 项目状态: 未结题
• 来源: https://www.nsf.gov/awardsearch/showAward?AWD_ID=1942926&HistoricalAwards=false
• 关键词: CAREER; Enabling; data; valuation; deletion

As data becomes an essential driver of technological and economic developments, it is critical to understand the value of data in different applications. This project develops a computational approach to quantify what type of data is more or less useful when the data is used to train prediction algorithms. This characterization of data value is important because it enables users to filter out poor quality data and to identify data that are important to collect in the future. In addition to data valuation, the project also develops complementary methods to facilitate deleting data from prediction algorithms. This would allow users to quickly remove poor quality data or data that might have privacy concerns from algorithms. Data valuation and data deletion are core aspects of recent policies aimed to enable individuals control over how their data is used and monetized by third-parties. The methods developed in this project can inform the implementation of such policies. This project develops a framework for data valuation based on extending the concept of Shapley value from economics. Shapley value measures how individual components contribute to the whole group. This project will build a rigorous statistical theory of data Shapley value, together with new scalable algorithms for estimating Shapley values on large datasets. Moreover, modifications to data Shapley value by relaxing its constraints will be investigated. Computing data Shapley value involves iteratively deleting certain data points and measuring the effect of this deletion on the performance of the trained machine learning model. This formulation closely links data valuation with the data deletion subproject. The goal of the latter is to efficiently delete subsets of the training data from a machine learning model without having to retrain from scratch. The data valuation and deletion methods will be implemented and validated on large publicly available biomedical datasets.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.

随着数据成为技术和经济发展的重要驱动力，理解数据在不同应用中的价值至关重要。该项目开发了一种计算方法来量化当数据用于训练预测算法时哪种类型的数据或多或少有用。数据价值的这种特征很重要，因为它使用户能够过滤掉质量差的数据，并确定将来需要收集的重要数据。除了数据评估，该项目还开发了辅助方法，以方便从预测算法中删除数据。这将允许用户从算法中快速删除质量差的数据或可能涉及隐私的数据。数据评估和数据删除是近期政策的核心内容，这些政策旨在使个人能够控制其数据如何被第三方使用和货币化。本项目制定的方法可以为这些政策的实施提供信息。本项目在扩展经济学中的Shapley值概念的基础上，开发了一个数据评估的框架。Shapley值衡量单个组件对整个组的贡献。该项目将建立一个严格的数据Shapley值统计理论，以及用于估计大型数据集上Shapley值的新的可扩展算法。此外，还将研究通过放宽数据Shapley值的约束来修改数据Shapley值。计算数据Shapley值涉及迭代删除某些数据点，并测量这种删除对训练后的机器学习模型性能的影响。这个公式将数据评估与数据删除子项目紧密联系在一起。后者的目标是从机器学习模型中有效地删除训练数据的子集，而无需从头开始重新训练。数据评估和删除方法将在大型公开生物医学数据集上实施和验证。该奖项反映了美国国家科学基金会的法定使命，并通过使用基金会的知识价值和更广泛的影响审查标准进行评估，被认为值得支持。

项目成果

Beta Shapley: a Unified and Noise-reduced Data Valuation Framework for Machine Learning

• 发表时间: 2021-10
• 作者: Yongchan Kwon;James Y. Zou

MetaShift: A Dataset of Datasets for Evaluating Contextual Distribution Shifts and Training Conflicts

• 发表时间: 2022-02
• 期刊: ArXiv
• 作者: Weixin Liang;James Y. Zou

Approximate Data Deletion from Machine Learning Models

机器学习模型中的近似数据删除

• 发表时间: 2021
• 期刊: AISTATS
• 作者: Izzo, Zach;Smart, Mary;Chaudhuri, Kamalika;Zou, James
• 通讯作者: Zou, James

MLDemon: Deployment Monitoring for Machine Learning Systems

• 发表时间: 2021-04
• 作者: Antonio A. Ginart;Martin Jinye Zhang;James Y. Zou

Efficient computation and analysis of distributional Shapley values

• 发表时间: 2020-07
• 作者: Yongchan Kwon;Manuel A. Rivas;James Y. Zou