Algorithmic Methods for Combining Expert Judgments

有关环境保护、国土安全以及其他关键问题的决策是通过数学模型来评估替代行动的风险和后果的。这些模型中的许多参数不能直接测量，因此模型严重依赖于专家的判断。近年来，已经发展了从专题专家小组中引出判断的正式结构化方法，并将其应用于重要的公共问题（例如，核电安全、空气污染）。在结构化专家判断中，每个专家以主观概率分布的形式报告他或她对参数值的信念。对于决策，将专家的判断组合成一个单一的概率分布是有用的。本文研究了备选组合规则的性能，包括简单平均、基于专家对真实值已知的变量的表现进行加权平均，以及根据贝叶斯规则更新决策者分布的方法。该研究通过使用综合专家判断来比较这些方法的性能，其中专家的数量和他们个人判断的质量（例如，精度，校准）是已知的。该项目还将使用获得的真实专家判断数据进行环境风险评估，以了解结果对替代组合方法和分析选择的敏感性。通过更好地理解使用专家判断的方法，包括涉及多少和如何多样化的一组专家以及如何综合他们的反应的问题，这项工作应该有助于更好的决策。

Decisions about environmental protection, homeland security, and other critical issues are informed by mathematical models used to estimate the risk and consequences of alternative actions. Many of the parameters in these models cannot be measured directly and so models rely heavily on expert judgment. In recent years, formal structured methods for eliciting judgments from a panel of subject-matter experts have been developed and applied to important public issues (e.g., nuclear power safety, air pollution). In structured expert judgment, each expert reports his or her beliefs about the value of a parameter as a subjective probability distribution. For decision making, it is useful to combine the experts' judgments into a single probability distribution. This research investigates the performance of alternative combination rules, including a simple average, a weighted average with weights based on the experts' performance on variables for which the true values become known, and a method that updates the decision maker's distribution in accordance with Bayes' Rule. The research compares the performance of these methods by using synthetic expert judgments in which the number of experts and the qualities of their individual judgments (e.g., precision, calibration) are known. The project will also use real expert-judgment data obtained for environmental risk assessment to see how sensitive results are to alternative combination methods and analytic choices. The work should contribute to better decision making through improved understanding of methods for using experts' judgments, including questions of how many and how diverse a set of experts to involve and how to synthesize their responses.