Doctoral Dissertation Research in DRMS: Developing and Validating a Method of Coherence-Based Judgment Aggregation

DRMS 博士论文研究：开发和验证基于一致性的判断聚合方法

• 批准号: 1919055
• 负责人: David Budescu
• 金额: $ 1.99万
• 依托单位: Fordham University
• 依托单位国家: 美国
• 项目类别: Standard Grant
• 财政年份: 2019
• 资助国家: 美国
• 起止时间: 2019-07-01 至 2021-06-30
• 项目状态: 已结题
• 来源: https://www.nsf.gov/awardsearch/showAward?AWD_ID=1919055&HistoricalAwards=false
• 关键词: Doctoral; Dissertation; Research; DRMS; Developing

High quality forecasting is essential to any area where people make decisions based on uncertain information, such as in geopolitics, the environment, and medical and financial domains. Subjective judgements of probabilities that quantify uncertainty are often used to produce such predictions. Previous research has differentiated two qualities of forecasts: accuracy, and coherence, or the extent of logical and probabilistic consistency in judgments. Recent work has suggested that coherent judgments tend to also be more accurate. Yet there exists no measure of coherence that might be used to identify a "select crowd" of judges who are more accurate than the average. The purpose of this research is to develop a psychological measure of coherence and determine how it predicts accuracy. Specifically, the research will compare the relative accuracy of judgments that are weighed by this new measure with judgments weighed by existing methods of judgment aggregation. Results of this work can help improve judgment quality in highly consequential fields. To answer this question, the researchers will create items that measure various aspects of probabilistic coherence. For example, items would test whether the probabilities assigned to all possible outcomes add to 1, as expected. Researchers will assess the psychometric properties - reliability, convergent and divergent validity - of the newly devised measure. To test the hypothesis that individuals with higher coherence scores are also more accurate, the researchers will weigh judgments from an incentivized forecasting tournament with this newly devised coherence measure. They will compare the accuracy of judgments weighed by this new method with other existing methods, such as linear and logarithmic averaging. The coherence measure may also produce a new, out-of-sample way of identifying the experts without additional knowledge of forecast performance, which will further our understanding of the role that individual-level coherence plays in maximizing judgment quality.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.

高质量的预测对于人们根据不确定信息做出决策的任何领域都至关重要，例如地缘政治，环境，医疗和金融领域。对量化不确定性的概率的主观判断通常用于产生这种预测。以前的研究区分了预测的两个质量：准确性和一致性，或判断中逻辑和概率一致性的程度。最近的研究表明，连贯的判断往往也更准确。然而，没有一种连贯性的衡量标准可用来确定比平均水平更准确的法官的“精选人群”。本研究的目的是发展一种连贯性的心理测量方法，并确定它如何预测准确性。 具体来说，研究将比较的相对准确性的判断，衡量了这种新的措施与判断加权现有的方法的判断聚合。这项工作的结果可以帮助提高判断质量在高度重要的领域。为了回答这个问题，研究人员将创建测量概率一致性各个方面的项目。 例如，项目将测试分配给所有可能结果的概率是否如预期的那样加为1。研究人员将评估新设计的测量方法的心理测量特性--可靠性、收敛和发散效度。 为了验证具有较高一致性得分的个体也更准确的假设，研究人员将使用这种新设计的一致性度量来衡量激励预测锦标赛的判断。他们将比较这种新方法与其他现有方法（如线性和对数平均）衡量的判断准确性。一致性措施也可能产生一种新的，样本外的方式来识别专家，而无需额外的知识预测性能，这将进一步我们的理解的作用，个人层面的一致性发挥最大限度地提高判断quality.This奖项反映了NSF的法定使命，并已被认为是值得通过使用基金会的智力价值和更广泛的影响审查标准进行评估的支持。