CHS: Small: Collaborative Research: Articulate+ - A Conversational Interface for Democr atizing Visual Analysis

CHS：小型：协作研究：Articulate - 用于民主化视觉分析的对话界面

• 批准号: 2007257
• 负责人: Andrew Johnson
• 金额: $ 33.3万
• 依托单位: University of Illinois at Chicago
• 依托单位国家: 美国
• 项目类别: Continuing Grant
• 财政年份: 2020
• 资助国家: 美国
• 起止时间: 2020-10-01 至 2023-09-30
• 项目状态: 已结题
• 来源: https://www.nsf.gov/awardsearch/showAward?AWD_ID=2007257&HistoricalAwards=false
• 关键词: CHS; Small; Collaborative; Research; Articulate+

The Internet gives the public, including scientists, access to vast amounts of data. Visualization is a powerful tool to take that data and turn it into a form that is easier to understand, helping people make better decisions. However, creating good visualizations from data is not easy. The Articulate+ Project will work to generate new knowledge about how to empower people with information visualization, by using speech and gesture as inputs, in new ways. A goal is to enable users to benefit from imprecise statements about what they want, rather than needing to give specific commands telling the computer what to do. This project will make it easier for citizens to engage in conversations with the computer about economic, medical data, demographic, transportation data, climate, and sports data; to answers that are more meaningful, and so to democratize data literacy. This research will use speech, gesture, and log data as new input interaction modalities for visual analytics. It will develop new understandings of how the imprecise and vague nature of natural language queries and gestures, as contextualized by work on a specific visual analysis problem, can be modeled as an opportunity for an intelligent software system to learn more about the underlying intent of the users. This model of intent, in turn, will be used to provide contextualized visualizations, which are expected to help those users gain valuable insights from their data. The audio speech data will be used to computationally model overhearing, to help infer users' current contextualized goals. The gesture data will be used to disambiguate expressions that refer to visual elements of a visualization (e.g., "that pie chart about electricity consumption"). The project will develop new understandings of how to combine the log of visualization state, the computational model of overhearing, and the gesture data in order to generate new visualizations in support of users’ work on tasks. Evaluations will be performed in both controlled laboratory and naturalistic study settings to determine whether effective semantic parsers can be derived for these specific visual analysis domains, and how this contextual interface affects users’ experience of visualization and discovery. The integrated annotated datasets from the studies will be made available to the research community satisfying a need for ecologically valid situated language interaction corpora.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.

互联网使包括科学家在内的公众能够获得大量的数据。可视化是一种强大的工具，可以将数据转化为更容易理解的形式，帮助人们做出更好的决策。然而，从数据中创建良好的可视化并不容易。Articulate+项目将致力于以新的方式，通过使用语音和手势作为输入，产生关于如何通过信息可视化赋予人们权力的新知识。一个目标是使用户能够从关于他们想要什么的不精确的陈述中受益，而不需要给出特定的命令告诉计算机做什么。该项目将使公民更容易与计算机进行有关经济，医疗数据，人口统计，交通数据，气候和体育数据的对话;更有意义的答案，从而使数据素养民主化。这项研究将使用语音，手势和日志数据作为视觉分析的新输入交互方式。它将发展新的理解，自然语言查询和手势的不精确和模糊的性质，作为一个特定的视觉分析问题的工作，可以建模为一个智能软件系统，以了解更多关于用户的潜在意图的机会。反过来，这种意图模型将用于提供情境化的可视化，预计将帮助这些用户从他们的数据中获得有价值的见解。音频语音数据将用于计算模型偷听，以帮助推断用户当前的上下文目标。手势数据将用于消除涉及可视化的视觉元素的表达的歧义（例如，“关于电力消耗的饼图”）。该项目将开发新的理解，如何结合可视化状态的日志，无意中听到的计算模型，和手势数据，以生成新的可视化，以支持用户的工作任务的联合收割机。将在受控实验室和自然主义研究环境中进行评估，以确定是否可以为这些特定的视觉分析域导出有效的语义解析器，以及这种上下文界面如何影响用户的可视化和发现体验。 从研究的综合注释数据集将提供给研究界满足生态有效的现场语言互动corpora.This奖项反映了NSF的法定使命的需要，并已被认为是值得通过使用基金会的智力价值和更广泛的影响审查标准进行评估的支持。

项目成果

An Investigation into an Always Listening Interface to Support Data Exploration

• DOI: 10.1145/3581641.3584079
• 发表时间: 2023-03
• 期刊: Proceedings of the 28th International Conference on Intelligent User Interfaces
• 作者: Roderick S. Tabalba;Nurit Kirshenbaum;J. Leigh;Abari Bhattacharya;Veronica Grosso;Barbara Di Eugenio;Andrew E. Johnson;Moira L. Zellner

Ditto: Exploring data in large display environments through speech+mid-air gesture interactions

同上：通过语音空中手势交互探索大型显示环境中的数据

• 发表时间: 2022
• 期刊: IEEE Vis 2022
• 作者: Aurisano, J.;Alsaiari, A.;Kumar, A.;Zellner, M;Leigh, J.;Di Eugenio, B.;Johnson, A.
• 通讯作者: Johnson, A.

Reference Resolution and New Entities in Exploratory Data Visualization: From Controlled to Unconstrained Interactions with a Conversational Assistant

• DOI: 10.18653/v1/2023.sigdial-1.33
• 发表时间: 2023
• 作者: Abari Bhattacharya;Abhinav Kumar;Barbara Di Eugenio;Roderick S. Tabalba;Jillian Aurisano;Veronica Grosso;Andrew Johnson;Jason Leigh;Moira L. Zellner