Efficient first-order predictor-corrector multiple objective optimization for fair misinformation detection-猫眼课题宝

DOI：

10.48550/arxiv.2209.07245

发表时间：

2022-09

期刊：

ArXiv

影响因子：

通讯作者：

中科院分区：

其他

文献类型：

作者：

Eric Enouen;Katja Mathesius;Sean Wang;Arielle K. Carr;Sihong Xie

多目标优化（MOO）旨在同时优化多个相互冲突的目标，并在机器学习中找到了重要的应用，例如最小化分类损失和处理不同群体公平性的差异。在最优状态下，进一步优化一个目标必然会损害至少另一个目标，决策者需要全面探索多个最优（称为帕累托前沿）以确定一个最终解决方案。我们解决了寻找帕累托前沿的效率问题。首先，对于大型神经网络和数据集，使用随机多梯度下降（SMGD）从头开始寻找前沿是昂贵的。我们建议基于预测校正方法，从几个初始最优点作为流形来探索帕累托锋面。其次，对于每个探索步骤，预测器求解一个大规模线性系统，该系统在模型参数数量上呈二次扩展，并且每次迭代求解器需要一次反向传播来评估二阶Hessian-vector积。我们提出了一个高斯-牛顿近似，它只线性扩展，并且每次迭代只需要一阶内积。当近似求解线性系统时，这也允许在MINRES和共轭梯度方法之间进行选择。这些创新使大型网络的预测校正成为可能。在多目标（公平性和准确性）错误信息检测任务上的实验表明：1)预测校正方法可以在更短的时间内找到优于或类似于SMGD的Pareto前沿；(2)一阶方法在不影响二阶方法识别的Pareto前沿质量的前提下，进一步缩短了运行时间。

Multiple-objective optimization (MOO) aims to simultaneously optimize multiple conflicting objectives and has found important applications in machine learning, such as minimizing classification loss and discrepancy in treating different populations for fairness. At optimality, further optimizing one objective will necessarily harm at least another objective, and decision-makers need to comprehensively explore multiple optima (called Pareto front) to pinpoint one final solution. We address the efficiency of finding the Pareto front. First, finding the front from scratch using stochastic multi-gradient descent (SMGD) is expensive with large neural networks and datasets. We propose to explore the Pareto front as a manifold from a few initial optima, based on a predictor-corrector method. Second, for each exploration step, the predictor solves a large-scale linear system that scales quadratically in the number of model parameters and requires one backpropagation to evaluate a second-order Hessian-vector product per iteration of the solver. We propose a Gauss-Newton approximation that only scales linearly, and that requires only first-order inner-product per iteration. This also allows for a choice between the MINRES and conjugate gradient methods when approximately solving the linear system. The innovations make predictor-corrector possible for large networks. Experiments on multi-objective (fairness and accuracy) misinformation detection tasks show that 1) the predictor-corrector method can find Pareto fronts better than or similar to SMGD with less time; and 2) the proposed first-order method does not harm the quality of the Pareto front identified by the second-order method, while further reduce running time.

Efficient first-order predictor-corrector multiple objective optimization for fair misinformation detection

文献摘要

相似文献