CRII: AF: Characterization and Complexity of Information Elicitation

人们判断预测的准确性的方式会极大地影响人们或计算机做出的预测。例如，格伦·布里尔在1950年辩称，对气象学家进行评估的方式实际上会促使他们扭曲真实的降雨概率。布里尔的研究激发了越来越多的统计学、经济学和现在的计算机科学领域的工作，这些科学研究的评估指标可以激励来自人或机器的准确报告。这些评估指标也被用于机器学习，机器学习是人工智能的一个分支，在机器学习中，设计师只提供评估指标本身，就会隐含地告诉计算机要预测什么统计数据。该项目力求从数学上描述统计数据和评价指标之间的这种联系，并进一步了解评价不同统计数据的计算和统计难度。对这一联系的准确理解将为新的评估指标提供潜在的潜力，以提高气候模拟和智能城市等大量应用的预测能力。特别是，量化不确定性或风险的统计指标可以改善许多领域的决策，包括医疗保健、工程和金融。机器学习中的一个主要算法范式，包括大多数回归技术和分类算法，是经验风险最小化(ERM)：根据称为损失函数的评估指标，从某个类别中选择最适合数据的模型。理论机器学习中的一条主线称为属性启发，它给出了一种数学形式来描述损失函数及其相应统计之间的联系。在这些术语中，本项目试图描述已校准损失函数的统计量，并确定需要多少回归参数或数据点才能保持校准。当将注意力限制在某些类别的损失函数时，这些问题与机器学习特别相关，这些损失函数可以很容易地优化或具有期望的统计学习保证。数理金融学中的一类统计数据被称为风险度量，用于监管银行，这构成了该项目的一个重要焦点。

The way one judges the accuracy of predictions can greatly impact what predictions people or computers make. For example, Glenn Brier argued in 1950 that the way meteorologists were evaluated would actually give them an incentive to distort the true probability of rain. Brier's study inspired a growing body of work in statistics, economics, and now computer science, which studies evaluation metrics that incentivize accurate reports from people or machines. These evaluation metrics are also used in machine learning, a branch of artificial intelligence, where a designer implicitly tells the computer what statistic to predict by providing only the evaluation metric itself. This project seeks to mathematically characterize this link between statistics and evaluation metrics, and moreover, to understand the computational and statistical difficulty of evaluating different statistics. A precise understanding of this link would provide new evaluation metrics with the potential to increase predictive power across a vast array of applications such as climate simulations and smart cities. In particular, metrics for statistics that quantify uncertainty or risk could improve decision making in many fields, including healthcare, engineering, and finance.A dominant algorithmic paradigm in machine learning, encompassing most regression techniques and classification algorithms, is that of empirical risk minimization (ERM): choosing a model from some class that best fits the data, according to some evaluation metric called a loss function. A thread of research in theoretical machine learning called property elicitation gives a mathematical formalism to describe the link between loss functions and their corresponding statistics. In these terms, this project seeks to characterize the statistics which have calibrated loss functions, and determine how many regression parameters or data points are required for the calibration to hold. These questions are particularly relevant to machine learning when restricting attention to certain classes of loss functions which can be easily optimized or which have desirable statistical learning guarantees. The class of statistics from mathematical finance known as risk measures, which are used to regulate banks, form an important focus of the project.