RI: Small: Robust Optimization of Loss Functions with Application to Active Learning

点击翻译按钮获取中文摘要

The goal of this project is to develop machine learning techniques that produce better predictions in a broad range of application domains where the usefulness of predictions is measured by application-specific performance measures. Existing machine learning methods are frequently forced to approximate these performance measures so that the search for a good predictor using the approximated measure will be efficient. This can produce inappropriate predictions for data in which the approximation to the performance measure is loose, even for the most fundamental performance measure: accuracy. The approach of this project instead approximates the training data and optimizes the exact performance measure to obtain a good predictor. Approximation takes the form of an "adversary" in a zero-sum game that chooses how the predicted variables are distributed for evaluation in a way that minimizes performance, but also matches properties of the dataset that are measured from training data. Many performance measures that are intractable to directly optimize become tractable when adversarially optimized. Resulting predictors are designed for the worst case and must perform at least as well when the adversary is replaced by real data with high probability.The societal impact of better aligning machine learning methods to a significantly wider range of performance measures is substantial. All classification and regression tasks that are currently solved using methods that approximate the desired performance measure, such as support vector machines or AdaBoost, could potentially be improved by the proposed approach. The project specifically investigates cost-sensitive classification, in which different mistakes incur penalties that are based on the implications of the prediction on real-world applications, F-measure maximization, which is a preferred performance measure balancing precision and recall in information retrieval tasks, and active learning, where the approach produces predictions that are robust to sample selection bias. Additional broader impacts of this project include developing new curriculum that will enable a wide range of data-driven practitioners to apply these improved methods to important application areas, including public policy, medical decision support, and epidemiology. Further, the PIs are committed to advising students from underrepresented groups at the University of Illinois at Chicago, which is an urban school with a diverse student population.