Doctoral Dissertation Research in DRMS: On the Evaluation of Beliefs: A Method for Assessing Credibility in Subjective Probability Judgment

不确定性是我们生活的世界的一个普遍特征。面对不确定性，决策者不能直接观察概率，而必须根据主观信念做出选择。这些信念通常被表达为主观概率判断（SPJ），其质量受到持有它们的个人的知识的限制。因此，在不确定性下的决策要求决策者考虑他们所掌握的判断的质量。在我们最关心的许多领域（例如，健康、安全和环境保护），然而，决策往往取决于单一事件的可能性，如手术的成功、共同基金的增长或极地冰盖的融化。因此，在许多决策领域，评估SPJ要么不切实际，要么不可能。为了避免这个问题，在不确定性下的判断研究往往被迫评估SPJ相对于（a）实验室的情况下，其中的基本利率或相对频率是已知的;（B）概率论要求一些定性排序或关系的判断集;或（c）观察到的不确定事件的结果。因此，虽然信息丰富，但对不确定性下判断的广泛研究对于现实世界的决策者在试图事先确定最佳行动方案时往往没有多大用处。本文提出了一种评估SPJ质量的方法，基础率、结果）未知。具体而言，本研究开发了一种方法，用于测量个人的SPJ倾向于同意优化的程度，聚合的“群众的智慧”，也被称为可信度。为了做到这一点，我们将进行几次概率预测比赛，并回归每个人的SPJ优化聚合，使用良好的判断项目开发的方法计算。这些模型的估计参数是个人的偏见指数;专业知识;和敏锐度（作为回归的标准误差）在主观概率判断。这项工作至少在两个方面改进了以前评估不确定性的方法。首先，这里开发的可信度框架可以用来评估SPJ，当规范性基准（例如，基本比率、结果、重复测量）不可用或未知。第二，初步证据表明，以这种方式建模的判断可以为决策者提供一种经验方法，用于纠正主观概率判断中的“错误”和“偏差”。此外，由于军事情报、气候科学和流行病学等领域的预测往往具有高度不确定性，而且错误的代价高得令人望而却步，因此这些领域的决策者在评估SPJ的能力方面甚至可以从边际收益中受益。目前的研究促进这种评价提供了直接的措施，一个源的质量，独立的经验结果。在这样做的过程中，这项研究提高了各种领域决策的质量和可评估性。

Uncertainty is a pervasive feature of the world we live in. In the face of uncertainty, decision makers cannot observe probabilities directly and must instead base their choices on subjective beliefs. These beliefs are often expressed as subjective probability judgments (SPJs), and their quality is constrained by the knowledge of the individuals who hold them. Consequently, decision making under uncertainty requires that decision makers account for the quality of the judgments at their disposal. In many of the areas that concern us the most (e.g., health, safety, and the protection of the environment), however, decisions often hinge on the likelihood of single events such as the success of a surgery, the growth of a mutual fund, or the melting of the polar ice caps. As a result, evaluating SPJs is either impractical or impossible in many areas of decision making. To circumvent this issue, research on judgment under uncertainty has often been forced to evaluate SPJs relative to (a) laboratory situations in which base rates or relative frequencies are known; (b) sets of judgments for which probability theory demands some qualitative ordering or relationship; or (c) the observed outcomes of uncertain events. Thus, while informative, the extensive body of research on judgment under uncertainty is often of little use to real-world decision makers when attempting to identify their best course of action, ex ante. This dissertation develops a method for assessing the quality of SPJs when benchmarks (e.g., base rates, outcomes) are unknown. Specifically, the present research develops a method for measuring the degree to which an individual's SPJs tend to agree with the optimized, aggregate "wisdom of the crowds", also known as credibility. To do this, we will conduct several probability forecasting tournaments and regress each individual's SPJs on optimized aggregates, calculated using methods developed by the Good Judgment Project. The estimated parameters of these models are indices of an individual's bias; expertise; and acuity (measured as the standard error of the regression) in subjective probability judgment. This work improves upon previous methods for assessing uncertainty in at least two ways. First, the credibility framework developed here can be used to evaluate SPJs when normative benchmarks (e.g., base rates, outcomes, repeated measurements) are unavailable or unknown. Second, preliminary evidence suggests that modeling judgments in this way can provide decision makers with an empirical method for correcting "errors" and "biases" in subjective probability judgment. Additionally, because predictions in domains such as military intelligence, climate science, and epidemiology tend be highly uncertain and errors prohibitively costly, decision makers in these domains may benefit from even marginal gains in their ability to evaluate SPJs. The present research facilitates this evaluation by providing straightforward measures of a source's quality, independent of empirical outcomes. In doing so, this research improves the quality and evaluability of decision making across a wide variety of domains.