Using case difficulty to improve predictive performance evaluation

点击翻译按钮获取中文摘要

In recent years, machine learning (ML) has led to a number of ground-breaking innovations in artificial intelligence (AI). In particular, predictive models have been at the forefront of this AI revolution (e.g., object detection in photos). Despite this seemingly substantial and rapid progress, some aspects of ML have not changed much for decades. One such area of stagnation is how all cases are treated equally in predictive performance evaluation. Most performance metrics are solely based on how many cases are predicted correctly or incorrectly and completely ignore which cases. As a result, the concept of case difficulty (i.e., how difficult is it to predict a given case correctly?) is not considered, often resulting in misleading performance evaluation. Although different cases tend to exhibit varying degrees of difficulty in many prediction problems, correctly/incorrectly predicting a difficult case is weighted just the same as correctly/incorrectly predicting an easy case. A related issue is whether all available test cases need to be used for predictive performance evaluation. Is it possible to accurately assess performance by only using a small number of test cases selected based on case difficulty? Inspired by how computerized tests such as the Graduate Record Examination (GRE) select and use only a subset of a pool of questions, this approach can prevent performance evaluation from being dominated by too easy or too difficult cases and mitigate insufficient data issues in many problem domains. My long-term research program focuses on developing novel ML methodologies that crosscut multiple problem domains. Over the next 5 years, my research will focus on developing novel predictive performance metrics and evaluation methods. The objectives are to: 1.Develop new predictive performance metrics based on case difficulty. These performance metrics will ensure correct prediction of difficult cases is rewarded, whereas incorrect prediction of easy cases is penalized. 2.Develop new case difficulty-based predictive performance evaluation methods that require fewer test cases than traditional methods. Test cases will be presented sequentially and the next test case will be selected based on case difficulty and whether the previous test cases were predicted correctly. The primary deliverable of this research will be an open-source Python package that implements our novel predictive performance metrics and evaluation methods. This package will enable ML researchers and engineers to incorporate case difficulty into their predictive modeling and precisely evaluate their models. In our increasingly AI-driven society, predictive models are more prevalent than ever. This research is important because it will lead to comprehensive insights into how these predictive models that affect our lives truly perform. Canadians will benefit from resulting trustworthy predictive models, further strengthening Canada's international leadership in ML research.