Using case difficulty to improve predictive performance evaluation

利用案例难度来改进预测绩效评估

• 批准号: RGPIN-2021-02588
• 负责人: Lee, Joon
• 金额: $ 2.11万
• 依托单位: University of Calgary
• 依托单位国家: 加拿大
• 项目类别: Discovery Grants Program - Individual
• 财政年份: 2021
• 资助国家: 加拿大
• 起止时间: 2021-01-01 至 2022-12-31
• 项目状态: 已结题
• 来源: https://www.nserc-crsng.gc.ca/ase-oro/Details-Detailles_eng.asp?id=742093
• 关键词: Using; case; difficulty; improve; predictive

In recent years, machine learning (ML) has led to a number of ground-breaking innovations in artificial intelligence (AI). In particular, predictive models have been at the forefront of this AI revolution (e.g., object detection in photos). Despite this seemingly substantial and rapid progress, some aspects of ML have not changed much for decades. One such area of stagnation is how all cases are treated equally in predictive performance evaluation. Most performance metrics are solely based on how many cases are predicted correctly or incorrectly and completely ignore which cases. As a result, the concept of case difficulty (i.e., how difficult is it to predict a given case correctly?) is not considered, often resulting in misleading performance evaluation. Although different cases tend to exhibit varying degrees of difficulty in many prediction problems, correctly/incorrectly predicting a difficult case is weighted just the same as correctly/incorrectly predicting an easy case. A related issue is whether all available test cases need to be used for predictive performance evaluation. Is it possible to accurately assess performance by only using a small number of test cases selected based on case difficulty? Inspired by how computerized tests such as the Graduate Record Examination (GRE) select and use only a subset of a pool of questions, this approach can prevent performance evaluation from being dominated by too easy or too difficult cases and mitigate insufficient data issues in many problem domains. My long-term research program focuses on developing novel ML methodologies that crosscut multiple problem domains. Over the next 5 years, my research will focus on developing novel predictive performance metrics and evaluation methods. The objectives are to: 1.Develop new predictive performance metrics based on case difficulty. These performance metrics will ensure correct prediction of difficult cases is rewarded, whereas incorrect prediction of easy cases is penalized. 2.Develop new case difficulty-based predictive performance evaluation methods that require fewer test cases than traditional methods. Test cases will be presented sequentially and the next test case will be selected based on case difficulty and whether the previous test cases were predicted correctly. The primary deliverable of this research will be an open-source Python package that implements our novel predictive performance metrics and evaluation methods. This package will enable ML researchers and engineers to incorporate case difficulty into their predictive modeling and precisely evaluate their models. In our increasingly AI-driven society, predictive models are more prevalent than ever. This research is important because it will lead to comprehensive insights into how these predictive models that affect our lives truly perform. Canadians will benefit from resulting trustworthy predictive models, further strengthening Canada's international leadership in ML research.

近年来，机器学习（ML）在人工智能（AI）领域带来了许多突破性的创新。特别是，预测模型一直处于这场人工智能革命的最前沿（例如，照片中的对象检测）。尽管取得了看似巨大而迅速的进展，但ML的某些方面几十年来并没有太大变化。其中一个停滞不前的领域是如何在预测性绩效评估中平等对待所有案例。大多数性能指标仅仅基于正确或错误预测的情况，而完全忽略了哪些情况。因此，案件难度的概念（即，正确预测一个给定的案例有多难？）没有考虑，往往导致误导性的业绩评价。虽然不同的情况下，往往表现出不同程度的困难，在许多预测问题，正确/错误地预测一个困难的情况下，加权只是正确/错误地预测一个简单的情况下。一个相关的问题是是否所有可用的测试用例都需要用于预测性能评估。是否有可能通过只使用少量的基于用例难度选择的测试用例来准确地评估性能？受研究生入学考试（GRE）等计算机化考试的启发，这种方法可以防止表现评估被太容易或太难的案例所主导，并减轻许多问题领域中的数据不足问题。我的长期研究计划专注于开发横切多个问题领域的新型ML方法。在接下来的5年里，我的研究将集中在开发新的预测性能指标和评估方法。其目标是：1.开发新的预测性能指标的基础上的情况下的困难。这些性能指标将确保对困难情况的正确预测得到奖励，而对简单情况的不正确预测受到惩罚。2.开发新的基于案例难度的预测性能评估方法，该方法需要比传统方法更少的测试案例。测试用例将按顺序呈现，下一个测试用例将根据用例难度和先前测试用例是否被正确预测来选择。这项研究的主要成果将是一个开源的Python包，它实现了我们新颖的预测性能指标和评估方法。该软件包将使机器学习研究人员和工程师能够将案例难度纳入其预测建模中，并精确评估其模型。在我们越来越多的人工智能驱动的社会中，预测模型比以往任何时候都更加普遍。这项研究很重要，因为它将导致对这些影响我们生活的预测模型如何真正执行的全面见解。加拿大人将受益于由此产生的值得信赖的预测模型，进一步加强加拿大在ML研究方面的国际领导地位。