CHS:Small: Incorporating and Balancing Stakeholder Values in Algorithm Design

该项目将为价值敏感型算法设计创建一种通用方法，并开发工具和技术，以帮助整合利益相关者的隐性价值观，平衡多个利益相关者的价值观，并在算法开发中实现集体目标。研究界越来越关注人的价值在算法设计和开发中的作用。例如，公平感知机器学习研究试图将公平概念转化为正式的算法约束，并开发受此类约束的算法。尽管这些方法具有数学上的严谨性，但先前的研究表明，当前具有歧视意识的机器学习研究与利益相关者的现实、背景和约束之间存在脱节；这种脱节可能会破坏实际行动。此外，研究表明，在与算法设计相关的各种值之间往往存在紧张关系。在重新设计维基百科的客观修订评估服务（ORES）的背景下，将开发一种新的通用方法，这是一种基于机器学习的服务，旨在生成编辑质量和文章质量的实时预测，这将使大量直接或间接通过其他应用程序消费维基百科内容的人受益。这项研究有四个主要目标。首先是阐明并演示一种创建尊重和平衡利益相关者价值的算法系统的一般方法。第二个目标是创建用于生成算法系统的价值报告和解释价值权衡的技术。第三个目标是创建、部署和评估社会和技术创新，以处理不同价值之间的基本权衡。最终目标是设计和实现对ORES的改进，这将改善依赖于ORES的各种应用程序，以及维基百科的内容和社区。举一个必须解决的问题的例子，质量控制算法优先考虑删除低质量内容的效率，这可能会破坏对等生产社区中贡献者的动机，特别是那些仍在学习如何贡献的新贡献者。然而，迄今为止，很少有工作被用于创建解决方案，以解决算法设计中不同值之间的紧张和权衡。该研究将通过多个研究来执行，通过逐步完成不同任务集的过程，每个任务都将允许与具有不同（可能是冲突）价值观的多个利益相关者进行交互。该奖项反映了美国国家科学基金会的法定使命，并通过使用基金会的知识价值和更广泛的影响审查标准进行评估，被认为值得支持。

This project will create a general method for value-sensitive algorithm design and develop tools and techniques to help incorporate the tacit values of stakeholders, balance multiple stakeholders' values, and achieve collective goals in the development of an algorithm. The research community has paid increasing attention to the role of human values in algorithm design and development. For example, fairness-aware machine learning research attempts to translate fairness notions into formal algorithmic constraints and develop algorithms subject to such constraints. Despite the mathematical rigor of these approaches, prior research suggests a disconnect between the current discrimination-aware machine learning research and stakeholders' realities, context, and constraints; this disconnect is likely to undermine practical initiatives. Furthermore, studies have suggested that there are often tensions among a diverse set of values relevant to the design of the algorithm. A new general method will be developed in the context of Redesigning Wikipedia's Objective Revision Evaluation Service (ORES), a machine learning-based service designed to generate real-time predictions on edit quality and article quality, which will benefit vast numbers of people who consume the Wikipedia content either directly or indirectly through other applications.There are four major goals of this research. The first is to articulate and demonstrate a general method for creating algorithmic systems that respect and balance stakeholders' values. The second goal is to create techniques for generating an algorithmic system's value report and explaining the value trade-offs. The third goal to create, deploy, and evaluate social and technical innovations to address fundamental trade-offs between different values. The final goal is to design and implement improvements to ORES, which will improve a wide variety of applications that rely on ORES, and Wikipedia's content and community as a whole. For an example of the kinds of problems that must be solved, quality control algorithms that prioritize efficiency in deleting low quality content incur the risk of undermining the motivation of contributors in peer production communities, particularly new contributors who are still learning how to contribute. To date, however, little work has been conducted to create solutions to address tensions and trade-offs between different values in algorithm design. The research will be performed through multiple studies by stepping through the process for a diverse set of tasks, each of which will allow interaction with multiple stakeholders, who have different (and perhaps conflicting) values.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.