RI: Small: Robust Optimization of Loss Functions with Application to Active Learning

该项目的目标是开发机器学习技术，在广泛的应用领域产生更好的预测，其中预测的有用性是通过特定于应用的性能衡量标准来衡量的。现有的机器学习方法经常被迫近似这些性能度量，因此使用近似度量搜索好的预测器将是有效的。这可能会产生对数据的不适当预测，在这些数据中，对性能衡量的近似是松散的，即使是对最基本的性能衡量：准确性也是如此。相反，该项目的方法是近似训练数据，并优化准确的性能度量，以获得良好的预测值。在零和博弈中，近似的形式是一个“对手”，它选择如何分配预测变量进行评估，以最小化性能，但也匹配从训练数据测量的数据集的属性。许多难以直接优化的性能指标在相反的优化时变得容易处理。由此产生的预测器是为最坏的情况而设计的，当对手被高概率的真实数据取代时，必须至少表现得同样好。更好地将机器学习方法与更广泛的性能测量范围相一致的社会影响是巨大的。目前使用接近期望性能度量的方法解决的所有分类和回归任务，例如支持向量机或AdaBoost，都有可能通过所提出的方法得到改进。该项目具体研究了成本敏感分类，在这种分类中，不同的错误会导致基于预测对现实世界应用的影响的惩罚；F-衡量最大化，这是在信息检索任务中平衡精确度和召回率的首选性能衡量标准；以及主动学习，其中该方法产生的预测对样本选择偏差是稳健的。该项目的其他更广泛的影响包括开发新的课程，使广泛的数据驱动从业者能够将这些改进的方法应用于重要的应用领域，包括公共政策、医疗决策支持和流行病学。此外，PI致力于为伊利诺伊大学芝加哥分校代表不足的学生提供建议，这是一所拥有多元化学生人口的城市学校。

The goal of this project is to develop machine learning techniques that produce better predictions in a broad range of application domains where the usefulness of predictions is measured by application-specific performance measures. Existing machine learning methods are frequently forced to approximate these performance measures so that the search for a good predictor using the approximated measure will be efficient. This can produce inappropriate predictions for data in which the approximation to the performance measure is loose, even for the most fundamental performance measure: accuracy. The approach of this project instead approximates the training data and optimizes the exact performance measure to obtain a good predictor. Approximation takes the form of an "adversary" in a zero-sum game that chooses how the predicted variables are distributed for evaluation in a way that minimizes performance, but also matches properties of the dataset that are measured from training data. Many performance measures that are intractable to directly optimize become tractable when adversarially optimized. Resulting predictors are designed for the worst case and must perform at least as well when the adversary is replaced by real data with high probability.The societal impact of better aligning machine learning methods to a significantly wider range of performance measures is substantial. All classification and regression tasks that are currently solved using methods that approximate the desired performance measure, such as support vector machines or AdaBoost, could potentially be improved by the proposed approach. The project specifically investigates cost-sensitive classification, in which different mistakes incur penalties that are based on the implications of the prediction on real-world applications, F-measure maximization, which is a preferred performance measure balancing precision and recall in information retrieval tasks, and active learning, where the approach produces predictions that are robust to sample selection bias. Additional broader impacts of this project include developing new curriculum that will enable a wide range of data-driven practitioners to apply these improved methods to important application areas, including public policy, medical decision support, and epidemiology. Further, the PIs are committed to advising students from underrepresented groups at the University of Illinois at Chicago, which is an urban school with a diverse student population.