Algorithmic Methods for Combining Expert Judgments

有关环境保护、国土安全和其他关键问题的决策是由数学模型提供信息的，这些模型用于估计替代行动的风险和后果。这些模型中的许多参数无法直接测量，因此模型严重依赖专家判断。近年来，已经开发出从专题专家小组获得判断的正式结构化方法，并将其应用于重要的公共问题(例如，核电安全、空气污染)。在结构化专家判断中，每个专家都以主观概率分布的形式报告他或她对参数值的信念。对于决策，将专家的判断组合成单一的概率分布是有用的。这项研究调查了可选组合规则的性能，包括简单平均、基于专家在真值已知的变量上的表现的加权平均以及根据贝叶斯规则更新决策者分布的方法。这项研究通过使用综合专家判断来比较这些方法的性能，在合成专家判断中，专家的数量和他们个人判断的质量(例如，精度、校准)是已知的。该项目还将使用为环境风险评估获得的真实专家判断数据，以了解结果对替代组合方法和分析选择的敏感性。这项工作应通过增进对使用专家判断的方法的理解，包括一组专家的参与人数和多样性以及如何综合他们的答复等问题，促进更好的决策。

Decisions about environmental protection, homeland security, and other critical issues are informed by mathematical models used to estimate the risk and consequences of alternative actions. Many of the parameters in these models cannot be measured directly and so models rely heavily on expert judgment. In recent years, formal structured methods for eliciting judgments from a panel of subject-matter experts have been developed and applied to important public issues (e.g., nuclear power safety, air pollution). In structured expert judgment, each expert reports his or her beliefs about the value of a parameter as a subjective probability distribution. For decision making, it is useful to combine the experts' judgments into a single probability distribution. This research investigates the performance of alternative combination rules, including a simple average, a weighted average with weights based on the experts' performance on variables for which the true values become known, and a method that updates the decision maker's distribution in accordance with Bayes' Rule. The research compares the performance of these methods by using synthetic expert judgments in which the number of experts and the qualities of their individual judgments (e.g., precision, calibration) are known. The project will also use real expert-judgment data obtained for environmental risk assessment to see how sensitive results are to alternative combination methods and analytic choices. The work should contribute to better decision making through improved understanding of methods for using experts' judgments, including questions of how many and how diverse a set of experts to involve and how to synthesize their responses.