RI: Small: Robust Optimization of Loss Functions with Application to Active Learning

该项目的目标是开发机器学习技术，在广泛的应用领域中产生更好的预测，其中预测的有用性是通过特定于应用程序的性能度量来衡量的。现有的机器学习方法经常被迫近似这些性能度量，以便使用近似度量来搜索良好的预测器将是有效的。这可能会对数据产生不适当的预测，其中对性能度量的近似是松散的，甚至对于最基本的性能度量：准确性也是如此。这个项目的方法反而接近训练数据并优化精确的性能度量来获得一个好的预测器。近似采用零和游戏中的“对手”的形式，选择如何以最小化性能的方式分配预测变量进行评估，但也匹配从训练数据中测量的数据集属性。许多难以直接优化的性能度量在对抗性优化后变得容易处理。结果预测器是为最坏的情况设计的，并且必须至少在对手被高概率的真实数据取代时表现良好。更好地将机器学习方法与更广泛的性能衡量标准结合起来，其社会影响是巨大的。所有分类和回归任务，目前都是使用近似所需性能度量的方法来解决的，例如支持向量机或AdaBoost，都可能通过本文提出的方法得到改进。该项目特别研究了成本敏感分类，其中不同的错误会导致基于预测对现实世界应用的影响的惩罚，F-measure最大化，这是在信息检索任务中平衡精度和召回率的首选性能指标，以及主动学习，其中该方法产生对样本选择偏差具有鲁棒性的预测。这个项目的其他更广泛的影响包括制定新的课程，使广泛的数据驱动的从业者能够将这些改进的方法应用于重要的应用领域，包括公共政策、医疗决策支持和流行病学。此外，私人顾问还致力于为伊利诺伊大学芝加哥分校（University of Illinois at Chicago）中代表性不足的群体的学生提供建议，这是一所学生群体多样化的城市学校。

The goal of this project is to develop machine learning techniques that produce better predictions in a broad range of application domains where the usefulness of predictions is measured by application-specific performance measures. Existing machine learning methods are frequently forced to approximate these performance measures so that the search for a good predictor using the approximated measure will be efficient. This can produce inappropriate predictions for data in which the approximation to the performance measure is loose, even for the most fundamental performance measure: accuracy. The approach of this project instead approximates the training data and optimizes the exact performance measure to obtain a good predictor. Approximation takes the form of an "adversary" in a zero-sum game that chooses how the predicted variables are distributed for evaluation in a way that minimizes performance, but also matches properties of the dataset that are measured from training data. Many performance measures that are intractable to directly optimize become tractable when adversarially optimized. Resulting predictors are designed for the worst case and must perform at least as well when the adversary is replaced by real data with high probability.The societal impact of better aligning machine learning methods to a significantly wider range of performance measures is substantial. All classification and regression tasks that are currently solved using methods that approximate the desired performance measure, such as support vector machines or AdaBoost, could potentially be improved by the proposed approach. The project specifically investigates cost-sensitive classification, in which different mistakes incur penalties that are based on the implications of the prediction on real-world applications, F-measure maximization, which is a preferred performance measure balancing precision and recall in information retrieval tasks, and active learning, where the approach produces predictions that are robust to sample selection bias. Additional broader impacts of this project include developing new curriculum that will enable a wide range of data-driven practitioners to apply these improved methods to important application areas, including public policy, medical decision support, and epidemiology. Further, the PIs are committed to advising students from underrepresented groups at the University of Illinois at Chicago, which is an urban school with a diverse student population.