Natural Language Interactions with Visual Analytics

自然语言与视觉分析的交互

• 批准号: RGPIN-2019-06340
• 负责人: Prince, MdEnamulHoque
• 金额: $ 2.4万
• 依托单位: York University
• 依托单位国家: 加拿大
• 项目类别: Discovery Grants Program - Individual
• 财政年份: 2022
• 资助国家: 加拿大
• 起止时间: 2022-01-01 至 2023-12-31
• 项目状态: 已结题
• 来源: https://www.nserc-crsng.gc.ca/ase-oro/Details-Detailles_eng.asp?id=757253
• 关键词: Natural; Language; Interactions; Visual; Analytics

Visual analytics has become a popular way for people to explore and understand data. It combines computational tools with information visualization techniques to facilitate the analytical reasoning process. While interaction is at the core of visual data analysis, the way people interact with data has largely remained the same over the years. Typically, people perform visual data analysis via mouse and touch interactions which are often inefficient when the interface requires many steps to complete a complex task. A promising way to support the user more effectively is to introduce natural language as a complementary input modality for visual analytics, as language can be more expressive and accessible compared to other modalities. The long-term goal of this research is to create the next generation of multimodal interfaces for visual analytics by incorporating natural language as an interaction modality which will enable people to perform complex data analysis tasks faster and more effectively. Using such futuristic interfaces, people will be able to express their complex information needs through a novel combination of input modalities including natural language, touch, gesture, and gaze, enabling rich interaction cycles with visual analytics systems. In response, the system will automatically generate answers combining visual representations with natural language explanations for effective comprehension. To this end, this proposal will focus on the following objectives. First, drawing from latest advances in automatic question answering research, we will develop methods for analyzing the syntactic, semantic, and pragmatic aspects of the question to determine what results should be presented to the user. Second, we will leverage natural language generation techniques to automatically create a multimedia summary by combining visualizations and text that effectively answers the question. Here, the text would not only facilitate users to comprehend the results more effectively by explaining key points but also enhance the transparency of the algorithm by conveying how the results were computed. Finally, we will investigate how combining natural language with other modalities affect the user while interacting with visual analytics and communicating insights. The research will open up new ways of analyzing large and complex datasets, improving the ability of Canadians to discover critical insights and make informed decisions in domains ranging from health informatics to business intelligence to customer analytics. The research program has great potential for supporting data democratization, enabling novice users and people who are visually impaired or who have cognitive difficulties to comprehend charts and perform analytical tasks through natural language. In addition to scientific impact, the program will provide specialized training for graduate and undergraduate students in data science and analytics for their future career.

可视化分析已成为人们探索和理解数据的流行方式。它将计算工具与信息可视化技术相结合，以促进分析推理过程。虽然交互是可视化数据分析的核心，但多年来人们与数据交互的方式基本保持不变。通常，人们通过鼠标和触摸交互来执行可视化数据分析，当界面需要许多步骤来完成复杂任务时，这些交互通常效率低下。更有效地支持用户的一种有希望的方法是引入自然语言作为视觉分析的补充输入模态，因为与其他模态相比，语言可以更具表达力和可访问性。这项研究的长期目标是通过将自然语言作为一种交互方式，为视觉分析创建下一代多模态界面，使人们能够更快，更有效地执行复杂的数据分析任务。 使用这种未来的界面，人们将能够通过包括自然语言、触摸、手势和凝视在内的输入方式的新颖组合来表达他们复杂的信息需求，从而实现与视觉分析系统的丰富交互周期。作为回应，系统将自动生成答案，将视觉表示与自然语言解释相结合，以实现有效的理解。为此，本提案将侧重于以下目标。首先，借鉴自动问答研究的最新进展，我们将开发分析问题的句法，语义和语用方面的方法，以确定应该向用户呈现什么样的结果。其次，我们将利用自然语言生成技术，通过结合可视化和文本来自动创建有效回答问题的多媒体摘要。在这里，文本不仅可以通过解释关键点来帮助用户更有效地理解结果，而且还可以通过传达结果是如何计算的来增强算法的透明度。最后，我们将研究自然语言与其他模态的结合如何影响用户，同时与可视化分析和交流见解进行交互。 该研究将开辟分析大型复杂数据集的新方法，提高加拿大人发现关键见解的能力，并在从健康信息学到商业智能再到客户分析等领域做出明智决策。该研究计划在支持数据民主化方面具有巨大的潜力，使新手用户和视力受损或有认知困难的人能够通过自然语言理解图表并执行分析任务。除了科学影响外，该计划还将为研究生和本科生提供数据科学和分析方面的专业培训，以促进他们未来的职业发展。