Natural Language Interactions with Visual Analytics

可视化分析已经成为人们探索和理解数据的一种流行方式。它结合了计算工具和信息可视化技术来促进分析推理过程。虽然交互是可视化数据分析的核心，但多年来人们与数据交互的方式在很大程度上保持不变。通常，人们通过鼠标和触摸交互来执行视觉数据分析，当界面需要许多步骤来完成复杂任务时，这种交互通常效率低下。一种更有效地支持用户的有希望的方法是引入自然语言作为视觉分析的补充输入方式，因为与其他方式相比，语言可以更具表现力和可访问性。本研究的长期目标是通过将自然语言作为一种交互方式，为视觉分析创建下一代多模式界面，使人们能够更快、更有效地执行复杂的数据分析任务。使用这种未来的界面，人们将能够通过自然语言、触摸、手势和凝视等输入方式的新颖组合来表达他们复杂的信息需求，从而实现与视觉分析系统的丰富交互周期。作为回应，系统将自动生成答案，将视觉表示与自然语言解释相结合，以实现有效理解。为此，本建议将着重于以下目标。首先，根据自动问答研究的最新进展，我们将开发分析问题的句法、语义和语用方面的方法，以确定应该向用户呈现什么样的结果。其次，我们将利用自然语言生成技术，通过结合可视化和有效回答问题的文本来自动创建多媒体摘要。在这里，文本不仅可以通过解释关键点来方便用户更有效地理解结果，还可以通过传达结果的计算方式来增强算法的透明度。最后，我们将研究如何将自然语言与其他模式相结合，在与视觉分析和交流见解交互的同时影响用户。该研究将开辟分析大型复杂数据集的新方法，提高加拿大人在健康信息学、商业智能和客户分析等领域发现关键见解并做出明智决策的能力。该研究计划在支持数据民主化方面具有巨大潜力，使新手用户和视障人士或有认知困难的人能够通过自然语言理解图表并执行分析任务。除了科学影响外，该项目还将为研究生和本科生提供数据科学和分析方面的专业培训，为他们未来的职业生涯做好准备。

Visual analytics has become a popular way for people to explore and understand data. It combines computational tools with information visualization techniques to facilitate the analytical reasoning process. While interaction is at the core of visual data analysis, the way people interact with data has largely remained the same over the years. Typically, people perform visual data analysis via mouse and touch interactions which are often inefficient when the interface requires many steps to complete a complex task. A promising way to support the user more effectively is to introduce natural language as a complementary input modality for visual analytics, as language can be more expressive and accessible compared to other modalities. The long-term goal of this research is to create the next generation of multimodal interfaces for visual analytics by incorporating natural language as an interaction modality which will enable people to perform complex data analysis tasks faster and more effectively. Using such futuristic interfaces, people will be able to express their complex information needs through a novel combination of input modalities including natural language, touch, gesture, and gaze, enabling rich interaction cycles with visual analytics systems. In response, the system will automatically generate answers combining visual representations with natural language explanations for effective comprehension. To this end, this proposal will focus on the following objectives. First, drawing from latest advances in automatic question answering research, we will develop methods for analyzing the syntactic, semantic, and pragmatic aspects of the question to determine what results should be presented to the user. Second, we will leverage natural language generation techniques to automatically create a multimedia summary by combining visualizations and text that effectively answers the question. Here, the text would not only facilitate users to comprehend the results more effectively by explaining key points but also enhance the transparency of the algorithm by conveying how the results were computed. Finally, we will investigate how combining natural language with other modalities affect the user while interacting with visual analytics and communicating insights. The research will open up new ways of analyzing large and complex datasets, improving the ability of Canadians to discover critical insights and make informed decisions in domains ranging from health informatics to business intelligence to customer analytics. The research program has great potential for supporting data democratization, enabling novice users and people who are visually impaired or who have cognitive difficulties to comprehend charts and perform analytical tasks through natural language. In addition to scientific impact, the program will provide specialized training for graduate and undergraduate students in data science and analytics for their future career.