Model Evaluation in Modern Predictive Regimes: Case Influence and Model Complexity

模型是科学研究的一个组成部分，它们可以在工程、商业和治理等许多实际领域中用作有效的设备。从数据中定义统计模型的方式已经取得了显着的进步，用于预测或准确描述我们观察到的世界。虽然复杂的预测模型经常出现，但我们对这些模型及其评估方法的理解一直滞后。非常需要评估模型和认识其缺陷的适当工具，以及说明其复杂性的适当方法。为了填补这些空白，该项目旨在开发方法和计算工具，用于评估预测环境中病例对一般模型的影响。它还将模型复杂性的概念扩展到一般的预测规则，通过使用整体模型对数据扰动的敏感性来进行模型比较和校准。本研究的成果不仅将通过精细化统计建模的实践为科学和工程带来巨大的好处，而且将通过应用为整个社会带来巨大的好处。特别是，该项目将在许多科学应用的异常值检测、许多商业应用的欺诈/威胁检测和预防以及人工智能应用的对抗性攻击检测方面具有实用价值。此外，它还将通过对模型复杂性的研究，促进我们对深度学习等现代算法模型的理解，并促进跨学科研究。该项目将为研究生提供研究培训机会。为了表征预测模型对数据的敏感性，PI将开发新的方法来评估一般建模程序的案例影响，包括许多现代统计学习技术，用于分类和回归。在线性回归中扩展案例删除统计和案例影响图，该项目将提供各种新的案例影响度量，特别是分类。此外，PI将开发有效的计算算法，通过利用同伦技术将两个建模问题与原始数据和各种扰动方案下的扰动数据联系起来，以评估这些情况的影响措施。此外，该项目将通过案例影响评估中考虑的模型对数据扰动的敏感性的透镜来检查模型的复杂性，并将模型自由度的概念扩展到包括大裕度分类器在内的一般建模程序。这一扩展将基于风险评估框架中的预期乐观和模型复杂性之间的关系，其中模型对个别情况扰动的敏感性可以与有条件的预期乐观联系起来。该奖项反映了NSF的法定使命，并被认为值得通过使用基金会的知识价值和更广泛的影响审查标准进行评估来支持。

Models are an integral component of scientific inquiries, and they can be used as effective devices in many practical areas as engineering, commerce, and governance. There have been remarkable advances in the way statistical models are defined from data for prediction or for accurate descriptions of the world we observe. While complex predictive models have routinely emerged, our understanding of these models and methods for their evaluation have been lagging. Appropriate tools for assessing models and recognizing their deficiencies, and proper ways to account for their complexity are in great need. To fill these gaps, the project aims to develop methodologies and computational tools for assessment of case influence on general models in predictive settings. And it will extend the notion of model complexity to general prediction rules for model comparison and calibration by using the overall model sensitivity to data perturbation. Results from this research will bring great benefit not only to science and engineering through the practice of refined statistical modeling, but also to society at large through applications. In particular, the project will have practical utility in outlier detection for many scientific applications, fraud/threat detection and prevention for many business applications, and detection of adversarial attacks for artificial intelligence applications. Moreover, it will advance our understanding of modern algorithmic models such as deep learning through the research on model complexity and foster interdisciplinary research. The project will provide research training opportunities for graduate students. Computational tools developed will be distributed as open-source software.To characterize the sensitivity of a predictive model to data, the PIs will develop novel approaches to case influence assessment for general modeling procedures, encompassing many modern statistical learning techniques for classification and regression. Extending case deletion statistics and case influence graph in linear regression, the project will offer a variety of new case influence measures for classification in particular. In addition, the PIs will develop efficient computational algorithms for evaluating those case influence measures by utilizing a homotopy technique to relate two modeling problems with the original data and perturbed data under various perturbation schemes. Further, the project will examine model complexity through the lens of model sensitivity to data perturbation considered in case influence assessment and extend the notion of model degrees of freedom to general modeling procedures including large-margin classifiers. This extension will be based on the relation between expected optimism and model complexity in the risk estimation framework where model sensitivities to perturbation of individual cases can be linked to the conditional expected optimism.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.