Natural Language Interactions with Visual Analytics

点击翻译按钮获取中文摘要

Visual analytics has become a popular way for people to explore and understand data. It combines computational tools with information visualization techniques to facilitate the analytical reasoning process. While interaction is at the core of visual data analysis, the way people interact with data has largely remained the same over the years. Typically, people perform visual data analysis via mouse and touch interactions which are often inefficient when the interface requires many steps to complete a complex task. A promising way to support the user more effectively is to introduce natural language as a complementary input modality for visual analytics, as language can be more expressive and accessible compared to other modalities. The long-term goal of this research is to create the next generation of multimodal interfaces for visual analytics by incorporating natural language as an interaction modality which will enable people to perform complex data analysis tasks faster and more effectively. Using such futuristic interfaces, people will be able to express their complex information needs through a novel combination of input modalities including natural language, touch, gesture, and gaze, enabling rich interaction cycles with visual analytics systems. In response, the system will automatically generate answers combining visual representations with natural language explanations for effective comprehension. To this end, this proposal will focus on the following objectives. First, drawing from latest advances in automatic question answering research, we will develop methods for analyzing the syntactic, semantic, and pragmatic aspects of the question to determine what results should be presented to the user. Second, we will leverage natural language generation techniques to automatically create a multimedia summary by combining visualizations and text that effectively answers the question. Here, the text would not only facilitate users to comprehend the results more effectively by explaining key points but also enhance the transparency of the algorithm by conveying how the results were computed. Finally, we will investigate how combining natural language with other modalities affect the user while interacting with visual analytics and communicating insights. The research will open up new ways of analyzing large and complex datasets, improving the ability of Canadians to discover critical insights and make informed decisions in domains ranging from health informatics to business intelligence to customer analytics. The research program has great potential for supporting data democratization, enabling novice users and people who are visually impaired or who have cognitive difficulties to comprehend charts and perform analytical tasks through natural language. In addition to scientific impact, the program will provide specialized training for graduate and undergraduate students in data science and analytics for their future career.