III: Medium: Collaborative Research: Closing the User-Model Loop for Understanding Topics in Large Document Collections

III：媒介：协作研究：关闭用户模型循环以理解大型文档集合中的主题

• 批准号: 1409287
• 负责人: Jordan Boyd-Graber
• 金额: $ 65万
• 依托单位: University of Maryland, College Park
• 依托单位国家: 美国
• 项目类别: Continuing Grant
• 财政年份: 2014
• 资助国家: 美国
• 起止时间: 2014-08-01 至 2020-07-31
• 项目状态: 已结题
• 来源: https://www.nsf.gov/awardsearch/showAward?AWD_ID=1409287&HistoricalAwards=false
• 关键词: III; Medium; Collaborative; Research; Closing

Individuals and organizations must cope with massive amounts of unstructured text information: individuals sifting through a lifetime of e-mail and documents, journalists understanding the activities of government organizations, companies reacting to what people say about them online, or scholars making sense of digitized documents from the ancient world. This project's research goal is to bring together two previously disconnected components of how users understand this deluge of data: algorithms to sift through the data and interfaces to communicate the results of the algorithms. This project will allow users to provide feedback to algorithms that were typically employed on a "take it or leave it" basis: if the algorithm makes a mistake or misunderstands the data, users can correct the problem using an intuitive user interface and improve the underlying analysis. This project will jointly improve both the algorithms and the interfaces, leading to deeper understanding of, faster introduction to, and greater trust in the algorithms we rely on to understand massive textual datasets. The resulting source code and functional demos will be broadly disseminated, and tutorials will be shared online and in person in educational efforts and to aid the adoption of the methodologies.This project enables computer algorithms and humans to apply their respective strengths and collaborate in managing and making sense of large volumes of textual data. It "closes the loop" in novel ways to connect users with a class of big data analysis algorithms called topic models. This connection is made through interfaces that empower the user to change the underlying models by refining the number and granularity of topics, adding or removing words considered by the model, and adding constraints on what words appear together in topics. The underlying model also enables new visualizations in the form of a Metadata Map that uses active learning to focus users' limited attention on the most important documents in a collection. Users annotate documents with useful meta-data and thereby further improve the quality of the discovered topics. The project includes evaluations of these methods through careful user studies and in-depth case studies to demonstrate that topics are more coherent, users can more quickly provide annotations, users trust the underlying algorithms more, and users can more effectively build an understanding of their textual data. The project web site (http://nlp.cs.byu.edu/closing-the-loop) will include pointers to the project Git repositories for source code, project demos, tutorials, and publications communicating experimental results.

个人和组织必须处理海量的非结构化文本信息：筛选一生的电子邮件和文件的个人，了解政府组织活动的记者，对人们在网上发表的评论做出反应的公司，或者理解古代世界数字化文件的学者。该项目的研究目标是将用户如何理解这一海量数据的两个以前互不相连的组成部分结合在一起：筛选数据的算法和交流算法结果的接口。该项目将允许用户向算法提供反馈，这些算法通常是在“接受或放弃”的基础上使用的：如果算法出错或误解了数据，用户可以使用直观的用户界面纠正问题，并改进潜在的分析。该项目将共同改进算法和界面，使我们对理解海量文本数据集所依赖的算法有更深入的理解、更快的介绍和更大的信任。由此产生的源代码和功能演示将被广泛传播，教程将在教育工作中在线和面对面分享，并帮助采用这些方法。该项目使计算机算法和人类能够发挥各自的优势，在管理和理解大量文本数据方面进行合作。它以一种新颖的方式将用户与一类名为主题模型的大数据分析算法联系起来。这种连接是通过界面实现的，这些界面使用户能够通过细化主题的数量和粒度、添加或删除模型考虑的单词以及添加对主题中一起出现的单词的约束来更改底层模型。底层模型还支持元数据映射形式的新可视化，该映射使用主动学习将用户有限的注意力集中在集合中最重要的文档上。用户用有用的元数据标注文档，从而进一步提高发现主题的质量。该项目包括通过仔细的用户研究和深入的案例研究对这些方法进行评估，以证明主题更连贯，用户可以更快地提供注释，用户更信任底层算法，用户可以更有效地建立对文本数据的理解。项目网站(http://nlp.cs.byu.edu/closing-the-loop)将包括指向项目Git存储库的指针，以获取源代码、项目演示、教程和交流实验结果的出版物。

项目成果

Which Evaluations Uncover Sense Representations that Actually Make Sense?

哪些评估揭示了真正有意义的意义表征？

• 发表时间: 2020
• 期刊: Proceedings of the 12th Language Resources and Evaluation Conference
• 作者: Jordan Boyd-Graber, Fenfei Guo

Why Didn’t You Listen to Me? Comparing User Control of Human-in-the-Loop Topic Models

• DOI: 10.18653/v1/p19-1637
• 发表时间: 2019-05
• 作者: Varun Kumar;Alison Smith-Renner;Leah Findlater;Kevin Seppi;Jordan L. Boyd-Graber

No Explainability without Accountability: An Empirical Study of Explanations and Feedback in Interactive ML

没有责任就没有可解释性：交互式机器学习中解释和反馈的实证研究

• DOI: 10.1145/3313831.3376624
• 发表时间: 2020
• 期刊: CHI '20: Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems
• 作者: Smith-Renner, Alison;Fan, Ron;Birchfield, Melissa;Wu, Tongshuang;Boyd-Graber, Jordan;Weld, Daniel S.;Findlater, Leah
• 通讯作者: Findlater, Leah

Automatic Evaluation of Local Topic Quality

• DOI: 10.18653/v1/p19-1076
• 发表时间: 2019-05
• 期刊: ArXiv
• 作者: Jeffrey Lund;Piper Armstrong;Wilson Fearn;Stephen Cowley;Courtni Byun;Jordan L. Boyd-Graber;Kevin Seppi

Digging into user control: perceptions of adherence and instability in transparent models

深入研究用户控制：透明模型中对依从性和不稳定性的看法

• DOI: 10.1145/3377325.3377491
• 发表时间: 2020
• 期刊: IUI '20: Proceedings of the 25th International Conference on Intelligent User Interfaces
• 作者: Smith-Renner, Alison;Kumar, Varun;Boyd-Graber, Jordan;Seppi, Kevin;Findlater, Leah
• 通讯作者: Findlater, Leah