Using case difficulty to improve predictive performance evaluation

近年来，机器学习(ML)在人工智能(AI)领域带来了许多突破性的创新。特别是，预测模型一直处于这场人工智能革命的前沿(例如，照片中的对象检测)。尽管取得了看似实质性和快速的进展，但ML的某些方面几十年来并没有太大变化。其中一个停滞不前的领域是如何在预测性业绩评估中平等对待所有案例。大多数性能指标完全基于预测正确或错误的案例数，而完全忽略了哪些案例。因此，案例难度的概念(即正确预测给定案例的难度有多大？)没有得到考虑，经常导致误导性的业绩评估。尽管在许多预测问题中，不同的情况往往表现出不同的困难程度，但正确/错误地预测困难的情况与正确/错误地预测简单的情况具有相同的权重。一个相关的问题是，是否所有可用的测试用例都需要用于预测性性能评估。仅仅使用基于用例难度选择的少量测试用例就可以准确地评估性能吗？受研究生入学考试(GRE)等计算机化考试的启发，这种方法可以防止绩效评估被太容易或太难的情况所主导，并缓解许多问题领域中数据不足的问题。我的长期研究计划专注于开发横跨多个问题领域的新型ML方法。在接下来的5年里，我的研究将集中在开发新的预测性绩效指标和评估方法上。目标是：1.根据案例难度设计新的预测性绩效指标。这些性能指标将确保对困难案例的正确预测得到奖励，而对简单案例的错误预测受到惩罚。2.提出了一种新的基于案例难度的预测性能评估方法，该方法比传统方法需要更少的测试用例。测试用例将按顺序呈现，并根据用例难度和先前测试用例是否被正确预测来选择下一个测试用例。这项研究的主要成果将是实现我们新的预测性能指标和评估方法的开源Python包。该软件包将使ML研究人员和工程师能够将案例难度纳入他们的预测建模中，并精确评估他们的模型。在我们日益由人工智能驱动的社会中，预测模型比以往任何时候都更加普遍。这项研究很重要，因为它将引导人们全面了解这些影响我们生活的预测模型是如何真正发挥作用的。加拿大人将受益于由此产生的值得信赖的预测模型，进一步加强加拿大在ML研究方面的国际领导地位。

In recent years, machine learning (ML) has led to a number of ground-breaking innovations in artificial intelligence (AI). In particular, predictive models have been at the forefront of this AI revolution (e.g., object detection in photos). Despite this seemingly substantial and rapid progress, some aspects of ML have not changed much for decades. One such area of stagnation is how all cases are treated equally in predictive performance evaluation. Most performance metrics are solely based on how many cases are predicted correctly or incorrectly and completely ignore which cases. As a result, the concept of case difficulty (i.e., how difficult is it to predict a given case correctly?) is not considered, often resulting in misleading performance evaluation. Although different cases tend to exhibit varying degrees of difficulty in many prediction problems, correctly/incorrectly predicting a difficult case is weighted just the same as correctly/incorrectly predicting an easy case. A related issue is whether all available test cases need to be used for predictive performance evaluation. Is it possible to accurately assess performance by only using a small number of test cases selected based on case difficulty? Inspired by how computerized tests such as the Graduate Record Examination (GRE) select and use only a subset of a pool of questions, this approach can prevent performance evaluation from being dominated by too easy or too difficult cases and mitigate insufficient data issues in many problem domains. My long-term research program focuses on developing novel ML methodologies that crosscut multiple problem domains. Over the next 5 years, my research will focus on developing novel predictive performance metrics and evaluation methods. The objectives are to: 1.Develop new predictive performance metrics based on case difficulty. These performance metrics will ensure correct prediction of difficult cases is rewarded, whereas incorrect prediction of easy cases is penalized. 2.Develop new case difficulty-based predictive performance evaluation methods that require fewer test cases than traditional methods. Test cases will be presented sequentially and the next test case will be selected based on case difficulty and whether the previous test cases were predicted correctly. The primary deliverable of this research will be an open-source Python package that implements our novel predictive performance metrics and evaluation methods. This package will enable ML researchers and engineers to incorporate case difficulty into their predictive modeling and precisely evaluate their models. In our increasingly AI-driven society, predictive models are more prevalent than ever. This research is important because it will lead to comprehensive insights into how these predictive models that affect our lives truly perform. Canadians will benefit from resulting trustworthy predictive models, further strengthening Canada's international leadership in ML research.