"You Can't Fix What You Can't Measure": Privately Measuring Demographic Performance Disparities in Federated Learning
"You Can't Fix What You Can't Measure": Privately Measuring Demographic Performance Disparities in Federated Learning
复制标题
DOI：
10.48550/arxiv.2206.12183
复制
发表时间：
2022-06
期刊：
影响因子：
--
通讯作者：
Marc Juárez;A. Korolova
中科院分区：
文献类型：
--
作者：
Marc Juárez;A. Korolova
与传统的机器学习模型一样，使用联合学习训练的模型可能会在不同的人口统计群体中表现出不同的性能。模特持有者必须识别这些差异，以减轻对群体的不必要伤害。然而，衡量模特在组中的表现需要访问关于组成员的信息，出于隐私原因，这些信息通常是有限的。我们提出了新的局部差异私有机制来衡量组之间的性能差异，同时保护组成员的隐私。为了分析这些机制的有效性，我们对它们在针对给定隐私预算进行优化时估计差异的误差进行了限制。我们的结果表明，对于实际数量的参与客户端，误差迅速减小，这表明，与先前的工作相反，保护隐私并不一定与识别联合模型的性能差异相冲突。
As in traditional machine learning models, models trained with federated learning may exhibit disparate performance across demographic groups. Model holders must identify these disparities to mitigate undue harm to the groups. However, measuring a model's performance in a group requires access to information about group membership which, for privacy reasons, often has limited availability. We propose novel locally differentially private mechanisms to measure differences in performance across groups while protecting the privacy of group membership. To analyze the effectiveness of the mechanisms, we bound their error in estimating a disparity when optimized for a given privacy budget. Our results show that the error rapidly decreases for realistic numbers of participating clients, demonstrating that, contrary to what prior work suggested, protecting privacy is not necessarily in conflict with identifying performance disparities of federated models.