Doctoral Dissertation Research in DRMS: Developing and Validating a Method of Coherence-Based Judgment Aggregation

对于人们根据不确定信息做出决策的任何领域，如地缘政治、环境、医疗和金融领域，高质量的预测都是必不可少的。对量化不确定性的概率的主观判断通常用于产生这种预测。先前的研究区分了预测的两种品质：准确性和连贯性，或判断中逻辑和概率一致性的程度。最近的研究表明，连贯的判断往往也更准确。然而，目前还没有一种一致性的衡量标准，可以用来确定比平均水平更准确的“精选人群”法官。本研究的目的是发展连贯的心理测量，并确定它如何预测准确性。具体来说，该研究将比较由这种新度量衡量的判断与由现有判断聚合方法衡量的判断的相对准确性。这项工作的结果可以帮助提高高度重要领域的判断质量。为了回答这个问题，研究人员将创建测量概率一致性各个方面的项目。例如，项目将测试分配给所有可能结果的概率是否如预期的那样加为1。研究人员将评估新设计的测量方法的心理测量特性——信度、收敛效度和发散效度。为了验证连贯性得分高的人也更准确的假设，研究人员将用这种新设计的连贯性测量来衡量激励预测锦标赛的判断。他们将比较这种新方法与其他现有方法（如线性平均和对数平均）的判断准确性。一致性度量也可以产生一种新的、样本外的方法来识别专家，而不需要额外的预测性能知识，这将进一步加深我们对个人层面的一致性在最大化判断质量中所起作用的理解。该奖项反映了美国国家科学基金会的法定使命，并通过使用基金会的知识价值和更广泛的影响审查标准进行评估，被认为值得支持。

High quality forecasting is essential to any area where people make decisions based on uncertain information, such as in geopolitics, the environment, and medical and financial domains. Subjective judgements of probabilities that quantify uncertainty are often used to produce such predictions. Previous research has differentiated two qualities of forecasts: accuracy, and coherence, or the extent of logical and probabilistic consistency in judgments. Recent work has suggested that coherent judgments tend to also be more accurate. Yet there exists no measure of coherence that might be used to identify a "select crowd" of judges who are more accurate than the average. The purpose of this research is to develop a psychological measure of coherence and determine how it predicts accuracy. Specifically, the research will compare the relative accuracy of judgments that are weighed by this new measure with judgments weighed by existing methods of judgment aggregation. Results of this work can help improve judgment quality in highly consequential fields. To answer this question, the researchers will create items that measure various aspects of probabilistic coherence. For example, items would test whether the probabilities assigned to all possible outcomes add to 1, as expected. Researchers will assess the psychometric properties - reliability, convergent and divergent validity - of the newly devised measure. To test the hypothesis that individuals with higher coherence scores are also more accurate, the researchers will weigh judgments from an incentivized forecasting tournament with this newly devised coherence measure. They will compare the accuracy of judgments weighed by this new method with other existing methods, such as linear and logarithmic averaging. The coherence measure may also produce a new, out-of-sample way of identifying the experts without additional knowledge of forecast performance, which will further our understanding of the role that individual-level coherence plays in maximizing judgment quality.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.