ITR/PE/SY(SBE): Dialogue-Assisted Visual Environment for Geoinformation: Enabling Collaborative Information Access and Decision-Making Through a Natural, Multimodal Interface

当前的计算系统不能有效地支持人类的工作。它们将人机交互限制在一次一种模式（通过键盘或鼠标），并且在设计时假设个人（而不是群体）使用，指导（而不是与系统交互）。为了支持人类工作和交互的方式，需要一种新的计算范式，它是多模态的，而不是单模态的；是协作的，而不是个人的；是对话的，而不是单向的。这项研究将开发原则，实施和评估自然，多模式，多用户，对话启用接口的地理信息系统（GIS），利用大屏幕显示和虚拟环境技术。这些界面将支持人与机器之间的双向迭代对话，其中人类参与者可以以自然的方式使用三种方式，语音，手势和凝视。向多模态人机界面的发展既需要对个体模态的理解，也需要对不同层次的信息进行融合。该项目特别关注使用计算机视觉和语音处理作为一种解释和整合来自口语、自由手势和凝视的信息的手段。它还涉及如何在地理信息系统的背景下与交互式多层地图进行人机对话，以及如何在人类合作者之间实现地图介导的对话。这项工作将通过使用一个现有的测试台来启动，该测试台集成了手势和语音，用于简单的地图查询。该研究将调查任务、信息和用户的特征如何影响人机和计算机介导的人机对话的性质。在这样做的过程中，研究将确定这些因素如何影响所需的战略，以实现一个有效的易于使用的界面，支持个人和团体获取信息和进行生产性工作。这项研究将推进自然、多模态人机交互科学，同时直接解决开发新知识的挑战，通过这些新知识，信息技术的进步可以使地理空间信息访问和应用的普遍参与成为可能。据估计，多达80%的数字数据包含某种形式的地理参考（坐标、地址、邮政编码）。因此，GIS已成为管理和分析科学、政策分析、资源管理、商业规划和教育应用等领域数据的重要工具。尽管地理信息系统的潜在应用和地理空间数据的日益可用性，地理信息系统仍未得到充分利用，因为目前的系统难以使用。目前的地理信息系统应用与地理信息系统促进公众获取大量地理空间信息以及支持公众和专家对影响其社区和环境的重要决策的承诺之间存在着很大的差距。因此，实现概述的研究目标将能够就公共安全和健康风险作出更有效的决策，并将支持更广泛的参与者为重要的公共决策提供投入。

Current computing systems do not support human work effectively. They restrict human-computer interaction to one mode at a time (through a keyboard or a mouse) and are designed with an assumption that use will be by individuals (rather than groups), directing (rather than interacting) with the system. To support the ways in which humans work and interact, a new paradigm for computing is required that is multimodal, rather than unimodal, collaborative, rather than personal, and dialogue-enabled, rather than unidirectional. This research will develop principles for, implement, and assess natural, multimodal, multiuser, dialogue-enabled interfaces to geographic information systems (GIS) that make use of large-screen displays and virtual environment technology. These interfaces will support a two-way iterative dialogue between human and machine in which the human participant can use three modalities, speech, gesture, and gaze, in natural ways. Progress toward multimodal human-computer interfaces will require both an understanding of individual modalities and the fusion of information at various levels. The project is concerned, specifically, with the use of computer vision and speech processing as a means of interpreting and integrating information from spoken words, free hand gestures and gaze. It is also concerned with how to enable a human-computer dialogue with an interactive, multi-layered map in the context of a GIS and with how to enable map-mediated dialogue between human collaborators. The work will be bootstrapped through use of an existing test-bed that integrates gesture and speech for simple map queries. The research will investigate how characteristics of tasks, information, and users influence the nature of human-computer and computer-mediated human-human dialogue. In doing so, the research will determine how these factors influence the strategies required to achieve an effective easy-to-use interface that supports information access and productive work by individuals and groups.This research will advance the science of natural, multimodal human-computer interaction, while addressing, directly, the challenges of developing new knowledge through which advances in information technology can enable universal participation in geospatial information access and applications. Estimates suggest that as much as 80% of all digital data contain some form of georeferencing (coordinates, addresses, zip codes). As a result, GIS have become an important tool for managing and analyzing data across a range of science, policy analysis, resource management, business planning, and educational applications. In spite of the potential applications for GIS and the growing availability of geospatial data, GIS are underutilized because current systems are hard to use. There is a substantial gap between current GIS applications and the promise of GIS to facilitate public access to the large volume of geospatial information and to support public as well as expert input to important decisions affecting their communities and environments. Achieving the research goals outlined will, thus, enable more effective decision-making about risks to public safety and health and will support input to important public decisions by a wider range of participants.