Data-driven selection of a convex loss function via shape-constrained estimation

这个研究项目的重点是损失函数的概念，这是机器学习和统计学的核心。损失函数测量模型预测的输出和实际输出之间的差异，它们通常满足称为凸性的特性，因此可以很容易地优化。损失函数量化了模型在描述数据时的准确性，因此，几乎所有的预测模型都是通过学习最小化给定损失函数的模型参数来计算的。选择一个好的损失函数是至关重要的；一个好的损失函数不仅可以改善我们的预测，还可以让我们建立更紧密的置信区间，并为我们提供对异常值更强的鲁棒性。虽然有选择合适损失函数的一般准则，但这些准则是定性的和不精确的；大多数人仍然默认使用一些标准选择，比如平方误差损失。这个项目的目标是开发方法来估计一个最优的凸损失函数从手头的数据。我们将设计、实现和测试算法，从业者可以使用这些算法自动获得专门针对其数据集优化的损失函数，这将使从业者能够做出更好的预测模型。这个项目的成功实施将对数据科学的标准实践产生深远的影响。该项目将与计划的本科和研究生教育组成部分深度整合。该项目的第一个组成部分将研究线性回归，并表明我们可以通过在称为Fisher散度的分布距离中近似具有对数凹密度的未知噪声分布来学习数据驱动的凸损失函数。该方法计算简单，在非高斯噪声的情况下，在估计精度、推理质量和鲁棒性方面显著提高了传统的平方误差损失。该项目的第二个组成部分将把这个想法扩展到多任务回归的设置，其中响应是多元的。该项目的第三个组成部分将分析分数匹配的理论性质，该统计方法支持凸损失估计的前两个组成部分，并且在统计学习的各种其他应用中具有重要意义。该奖项反映了美国国家科学基金会的法定使命，并通过使用基金会的知识价值和更广泛的影响审查标准进行评估，被认为值得支持。

This research project focuses on the notion of loss functions, which is central to machine learning and statistics. Loss functions measure the difference between the output predicted by the model and the actual output, and they typically satisfy a property called convexity so that they can be easily optimized. Loss functions quantify how accurate a model is at describing the data and therefore, almost all predictive models are computed by learning model parameters which minimize a given loss function. Choosing a good loss function is vitally important; a good loss function not only improves our predictions, but also allows us to build tighter confidence intervals, and gives us greater robustness to outliers. Although there are general guidelines for choosing a suitable loss function, these guidelines are qualitative and imprecise; most people still default to a few standard choices such as the square error loss. The goal of this project is to develop methods to estimate an optimal convex loss function from the data at hand. We will design, implement, and test algorithms that practitioners can use to automatically obtain loss functions specifically optimized to their dataset, which will allow the practitioners to make better predictive models. Successful execution of this project will have far-reaching effects on standard practices in data science. This project will be deeply integrated with the planned educational components at both the undergraduate and graduate levels.The first component of the project will look at linear regression and show that we can learn a data-driven convex loss function by approximating the unknown noise distribution with a log-concave density in a distributional distance known as the Fisher divergence. The proposed approach is computationally simple and, in settings where the noise is non-Gaussian, significantly improves upon the traditional squared error loss in estimation accuracy, inference quality, and robustness. The second component of the project will extend the idea to the setting of multi-task regression where the response is multivariate. The third component of the project will analyze the theoretical properties of score matching–the statistical method that underpins the first two components on convex loss estimation as well as being of fundamental importance in various other applications in statistical learning.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.