Help or Hinder? Evaluating the Impact of Fairness Metrics and Algorithms in Visualizations for Consensus Ranking
Help or Hinder? Evaluating the Impact of Fairness Metrics and Algorithms in Visualizations for Consensus Ranking
复制标题
DOI：
10.1145/3593013.3594108
复制
发表时间：
2023-06
期刊：
影响因子：
--
通讯作者：
Hilson Shrestha;Kathleen Cachel;Mallak Alkhathlan;Elke A. Rundensteiner;Lane Harrison
中科院分区：
文献类型：
--
作者：
Hilson Shrestha;Kathleen Cachel;Mallak Alkhathlan;Elke A. Rundensteiner;Lane Harrison
对于多个利益攸关方提供建议的应用，公平的共识排名不仅必须确保排名者的偏好得到很好的代表，而且还必须在最终结果中减轻社会人口群体中的不利因素。然而，对于将公平指标和算法可视化并将其集成到人在环系统中以帮助决策者的价值或挑战，几乎没有经验指导。在这项工作中，我们设计了一项研究来分析基于公平性度量的可视化和算法集成的有效性。我们通过一个基于任务的众包实验来探索这一点，该实验比较了用于构建共识排名的交互式可视化系统ConensusFuse和类似的系统，该系统包括公平度量和公平排名生成算法的可视编码。我们分析了在构建这两个系统的公平共识排名过程中的公平性度量、排名者决策的一致性以及用户交互。在我们对200名参与者的研究中，结果表明，提供这些面向公平的支持功能可以推动用户将他们的决定与公平衡量标准保持一致，同时最大限度地减少手动修改共识排名的繁琐过程。我们讨论了这些结果对下一代面向公平的系统设计的影响，以及未来研究的新方向。
For applications where multiple stakeholders provide recommendations, a fair consensus ranking must not only ensure that the preferences of rankers are well represented, but must also mitigate disadvantages among socio-demographic groups in the final result. However, there is little empirical guidance on the value or challenges of visualizing and integrating fairness metrics and algorithms into human-in-the-loop systems to aid decision-makers. In this work, we design a study to analyze the effectiveness of integrating such fairness metrics-based visualization and algorithms. We explore this through a task-based crowdsourced experiment comparing an interactive visualization system for constructing consensus rankings, ConsensusFuse, with a similar system that includes visual encodings of fairness metrics and fair-rank generation algorithms, FairFuse. We analyze the measure of fairness, agreement of rankers’ decisions, and user interactions in constructing the fair consensus ranking across these two systems. In our study with 200 participants, results suggest that providing these fairness-oriented support features nudges users to align their decision with the fairness metrics while minimizing the tedious process of manually having to amend the consensus ranking. We discuss the implications of these results for the design of next-generation fairness oriented-systems and along with emerging directions for future research.