Doctoral Dissertation Research in DRMS: On the Evaluation of Beliefs: A Method for Assessing Credibility in Subjective Probability Judgment

不确定性是我们生活的世界的一个普遍特征。面对不确定性，决策者不能直接观察概率，而必须基于主观信念做出选择。这些信念通常被表达为主观概率判断(SPJ)，其质量受到持有这些信念的个人的知识的限制。因此，在不确定的情况下进行决策要求决策者对其可支配的判决的质量作出解释。然而，在许多我们最关心的领域(例如，健康、安全和环境保护)，决策往往取决于单一事件的可能性，如手术成功、共同基金的增长或极地冰盖的融化。因此，在许多决策领域，评估SPJ要么不切实际，要么不可能。为了规避这一问题，对不确定情况下的判断的研究往往被迫评估与(A)基本比率或相对频率已知的实验室情况有关的SPJ；(B)概率论要求某种定性排序或关系的判断集合；或(C)不确定事件的观察结果。因此，虽然对不确定情况下的判断进行了广泛的研究，但在试图确定他们的最佳行动方案时，对现实世界的决策者往往用处不大。本文发展了一种在基准(例如，基本利率、结果)未知的情况下评估SPJ质量的方法。具体地说，这项研究开发了一种方法，用来衡量一个人的SPJ倾向于在多大程度上符合优化的、聚合的“群体智慧”，也就是众所周知的可信度。为了做到这一点，我们将进行几次概率预测锦标赛，并将每个人的SPJ回归到优化的聚集物上，这些聚集体使用Good判断项目开发的方法进行计算。这些模型的估计参数是个人在主观概率判断中的偏见、专长和敏锐度(以回归的标准误差衡量)的指数。这项工作至少在两个方面改进了以前的不确定性评估方法。首先，在无法获得或未知规范性基准(例如，基本比率、结果、重复测量)时，可使用此处制定的可信度框架来评价SPJ。第二，初步证据表明，以这种方式建模判断可以为决策者提供一种经验方法，以纠正主观概率判断中的“错误”和“偏差”。此外，由于军事情报、气候科学和流行病学等领域的预测往往高度不确定，错误代价高得令人望而却步，这些领域的决策者可能会受益于他们评估SPJ的能力的微乎其微的进步。本研究通过提供与经验结果无关的对来源质量的直截了当的测量来促进这一评估。通过这样做，这项研究提高了广泛领域决策的质量和可评估性。

Uncertainty is a pervasive feature of the world we live in. In the face of uncertainty, decision makers cannot observe probabilities directly and must instead base their choices on subjective beliefs. These beliefs are often expressed as subjective probability judgments (SPJs), and their quality is constrained by the knowledge of the individuals who hold them. Consequently, decision making under uncertainty requires that decision makers account for the quality of the judgments at their disposal. In many of the areas that concern us the most (e.g., health, safety, and the protection of the environment), however, decisions often hinge on the likelihood of single events such as the success of a surgery, the growth of a mutual fund, or the melting of the polar ice caps. As a result, evaluating SPJs is either impractical or impossible in many areas of decision making. To circumvent this issue, research on judgment under uncertainty has often been forced to evaluate SPJs relative to (a) laboratory situations in which base rates or relative frequencies are known; (b) sets of judgments for which probability theory demands some qualitative ordering or relationship; or (c) the observed outcomes of uncertain events. Thus, while informative, the extensive body of research on judgment under uncertainty is often of little use to real-world decision makers when attempting to identify their best course of action, ex ante. This dissertation develops a method for assessing the quality of SPJs when benchmarks (e.g., base rates, outcomes) are unknown. Specifically, the present research develops a method for measuring the degree to which an individual's SPJs tend to agree with the optimized, aggregate "wisdom of the crowds", also known as credibility. To do this, we will conduct several probability forecasting tournaments and regress each individual's SPJs on optimized aggregates, calculated using methods developed by the Good Judgment Project. The estimated parameters of these models are indices of an individual's bias; expertise; and acuity (measured as the standard error of the regression) in subjective probability judgment. This work improves upon previous methods for assessing uncertainty in at least two ways. First, the credibility framework developed here can be used to evaluate SPJs when normative benchmarks (e.g., base rates, outcomes, repeated measurements) are unavailable or unknown. Second, preliminary evidence suggests that modeling judgments in this way can provide decision makers with an empirical method for correcting "errors" and "biases" in subjective probability judgment. Additionally, because predictions in domains such as military intelligence, climate science, and epidemiology tend be highly uncertain and errors prohibitively costly, decision makers in these domains may benefit from even marginal gains in their ability to evaluate SPJs. The present research facilitates this evaluation by providing straightforward measures of a source's quality, independent of empirical outcomes. In doing so, this research improves the quality and evaluability of decision making across a wide variety of domains.