Judgment Sieve: Reducing Uncertainty in Group Judgments through Interventions Targeting Ambiguity versus Disagreement
Judgment Sieve: Reducing Uncertainty in Group Judgments through Interventions Targeting Ambiguity versus Disagreement
复制标题
DOI：
10.1145/3610074
复制
发表时间：
2023-05
影响因子：
--
通讯作者：
Quan Ze Chen;Amy X. Zhang
中科院分区：
文献类型：
--
作者：
Quan Ze Chen;Amy X. Zhang
当一群人被要求做出判断时，不确定性的问题经常出现。现有的减少不确定性的方法通常侧重于迭代地提高整个任务指令的特异性。然而，不确定性可能来自多个来源，例如由于有限的上下文而被判断的项目的模糊性，或者由于不同的观点和未指定的任务而导致参与者之间的分歧。如果没有针对不确定性的正确来源，一刀切的干预可能是无效的。本文引入了一种新的工作流程——判断筛，以有针对性地减少群体判断任务中的不确定性。通过在最初一轮判断引出中使用分离不同不确定性来源的测量，我们可以选择有针对性的干预，添加上下文或审议，以最有效地减少每个被判断项目的不确定性。我们在两个任务上测试了我们的方法：评价词对相似性和在线评论的毒性，表明有针对性的干预减少了大多数不确定情况下的不确定性。在前10%的案例中，我们看到两个任务的歧义度分别减少了21.4%和25.7%，歧义度分别减少了22.2%和11.2%。我们还通过模拟发现，我们的目标方法减少了两种不确定性来源的平均不确定性分数，而不是统一的方法，其中一个来源的平均不确定性的减少伴随着另一个来源的增加。
When groups of people are tasked with making a judgment, the issue of uncertainty often arises. Existing methods to reduce uncertainty typically focus on iteratively improving specificity in the overall task instruction. However, uncertainty can arise from multiple sources, such as ambiguity of the item being judged due to limited context, or disagreements among the participants due to different perspectives and an under-specified task. A one-size-fits-all intervention may be ineffective if it is not targeted to the right source of uncertainty. In this paper we introduce a new workflow, Judgment Sieve, to reduce uncertainty in tasks involving group judgment in a targeted manner. By utilizing measurements that separate different sources of uncertainty during an initial round of judgment elicitation, we can then select a targeted intervention adding context or deliberation to most effectively reduce uncertainty on each item being judged. We test our approach on two tasks: rating word pair similarity and toxicity of online comments, showing that targeted interventions reduced uncertainty for the most uncertain cases. In the top 10% of cases, we saw an ambiguity reduction of 21.4% and 25.7%, and a disagreement reduction of 22.2% and 11.2% for the two tasks respectively. We also found through a simulation that our targeted approach reduced the average uncertainty scores for both sources of uncertainty as opposed to uniform approaches where reductions in average uncertainty from one source came with an increase for the other.