Model Evaluation in Modern Predictive Regimes: Case Influence and Model Complexity

模型是科学研究不可或缺的组成部分，它们可以在工程、商业和治理等许多实际领域中用作有效的工具。在为预测或准确描述我们观察到的世界而从数据中定义统计模型的方式方面，已经取得了显着的进步。虽然复杂的预测模型经常出现，但我们对这些模型和评估方法的理解一直滞后。迫切需要适当的工具来评估模型并认识到它们的不足之处，并以适当的方式说明它们的复杂性。为了填补这些空白，该项目旨在开发方法和计算工具，用于在预测环境中评估病例对一般模型的影响。它将模型复杂性的概念扩展到一般预测规则，通过利用整体模型对数据扰动的敏感性来进行模型比较和校准。这项研究的结果不仅将通过精细化统计建模的实践为科学和工程带来巨大的好处，而且将通过应用为整个社会带来巨大的好处。特别是，该项目将在许多科学应用程序的离群值检测、许多商业应用程序的欺诈/威胁检测和预防以及人工智能应用程序的对抗性攻击检测方面具有实用价值。此外，通过对模型复杂性的研究，将促进我们对深度学习等现代算法模型的理解，并促进跨学科研究。该项目将为研究生提供研究培训机会。开发的计算工具将作为开放源码软件分发。为了表征预测模型对数据的敏感性，私人投资促进机构将为一般建模程序开发新的案例影响评估方法，包括许多用于分类和回归的现代统计学习技术。扩展线性回归中的案例删除统计和案例影响图，该项目将特别为分类提供各种新的案例影响度量。此外，PI将开发有效的计算算法来评估这些情况影响措施，利用同伦技术将两个建模问题与原始数据和各种扰动方案下的扰动数据联系起来。此外，该项目将通过案例影响评估中考虑的模型对数据扰动的敏感性的透镜来审查模型的复杂性，并将模型自由度的概念扩展到一般建模程序，包括大范围分类器。这一延期将基于风险估计框架中预期乐观与模型复杂性之间的关系，其中模型对个案扰动的敏感性可与有条件的预期乐观联系在一起。该奖项反映了NSF的法定使命，并已通过使用基金会的智力优势和更广泛的影响审查标准进行评估，被认为值得支持。

Models are an integral component of scientific inquiries, and they can be used as effective devices in many practical areas as engineering, commerce, and governance. There have been remarkable advances in the way statistical models are defined from data for prediction or for accurate descriptions of the world we observe. While complex predictive models have routinely emerged, our understanding of these models and methods for their evaluation have been lagging. Appropriate tools for assessing models and recognizing their deficiencies, and proper ways to account for their complexity are in great need. To fill these gaps, the project aims to develop methodologies and computational tools for assessment of case influence on general models in predictive settings. And it will extend the notion of model complexity to general prediction rules for model comparison and calibration by using the overall model sensitivity to data perturbation. Results from this research will bring great benefit not only to science and engineering through the practice of refined statistical modeling, but also to society at large through applications. In particular, the project will have practical utility in outlier detection for many scientific applications, fraud/threat detection and prevention for many business applications, and detection of adversarial attacks for artificial intelligence applications. Moreover, it will advance our understanding of modern algorithmic models such as deep learning through the research on model complexity and foster interdisciplinary research. The project will provide research training opportunities for graduate students. Computational tools developed will be distributed as open-source software.To characterize the sensitivity of a predictive model to data, the PIs will develop novel approaches to case influence assessment for general modeling procedures, encompassing many modern statistical learning techniques for classification and regression. Extending case deletion statistics and case influence graph in linear regression, the project will offer a variety of new case influence measures for classification in particular. In addition, the PIs will develop efficient computational algorithms for evaluating those case influence measures by utilizing a homotopy technique to relate two modeling problems with the original data and perturbed data under various perturbation schemes. Further, the project will examine model complexity through the lens of model sensitivity to data perturbation considered in case influence assessment and extend the notion of model degrees of freedom to general modeling procedures including large-margin classifiers. This extension will be based on the relation between expected optimism and model complexity in the risk estimation framework where model sensitivities to perturbation of individual cases can be linked to the conditional expected optimism.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.