ITR/PE/SY(SBE): Dialogue-Assisted Visual Environment for Geoinformation: Enabling Collaborative Information Access and Decision-Making Through a Natural, Multimodal Interface

当前的计算系统不能有效地支持人类工作。它们将人机交互限制在一次一种模式(通过键盘或鼠标)，并在设计时假设使用将由个人(而不是群体)使用，指导(而不是交互)系统。为了支持人类工作和互动的方式，需要一种新的计算范式，它是多模式的，而不是单峰的，协作的，而不是个人的，支持对话的，而不是单向的。这项研究将为利用大屏幕显示和虚拟环境技术的地理信息系统(GIS)开发、实施和评估自然的、多模式的、多用户的、支持对话的界面。这些界面将支持人与机器之间的双向迭代对话，其中人类参与者可以自然地使用三种方式：语音、手势和凝视。在实现多模式人机界面方面取得进展，既需要了解个别模式，也需要融合不同层次的信息。具体而言，该项目涉及使用计算机视觉和语音处理作为一种手段来解释和整合来自口语、手势和凝视的信息。它还涉及如何在地理信息系统的背景下使用交互式、多层地图实现人机对话，以及如何实现人类合作者之间的地图中介对话。这项工作将通过使用现有的集成手势和语音的试验台来完成，以进行简单的地图查询。这项研究将调查任务、信息和用户的特征如何影响人机和计算机中介的人与人对话的性质。在此过程中，研究将确定这些因素如何影响实现有效、易于使用的界面所需的战略，该界面支持个人和群体获取信息和进行生产性工作。这项研究将推进自然、多模式人机交互的科学，同时直接解决开发新知识的挑战，通过这些新知识，信息技术的进步可以使普遍参与地理空间信息获取和应用。据估计，多达80%的数字数据包含某种形式的地理参考(坐标、地址、邮政编码)。因此，地理信息系统已成为管理和分析各种科学、政策分析、资源管理、业务规划和教育应用程序中数据的重要工具。尽管地理信息系统的潜在应用和地理空间数据的可用性不断提高，但由于目前的系统很难使用，地理信息系统仍未得到充分利用。目前的地理信息系统应用与地理信息系统在便利公众获取大量地理空间信息和支持公众以及专家对影响其社区和环境的重要决策的投入方面的承诺之间存在很大差距。因此，实现概述的研究目标将使对公共安全和健康的风险作出更有效的决策，并将支持更广泛的参与者对重要公共决策的投入。

Current computing systems do not support human work effectively. They restrict human-computer interaction to one mode at a time (through a keyboard or a mouse) and are designed with an assumption that use will be by individuals (rather than groups), directing (rather than interacting) with the system. To support the ways in which humans work and interact, a new paradigm for computing is required that is multimodal, rather than unimodal, collaborative, rather than personal, and dialogue-enabled, rather than unidirectional. This research will develop principles for, implement, and assess natural, multimodal, multiuser, dialogue-enabled interfaces to geographic information systems (GIS) that make use of large-screen displays and virtual environment technology. These interfaces will support a two-way iterative dialogue between human and machine in which the human participant can use three modalities, speech, gesture, and gaze, in natural ways. Progress toward multimodal human-computer interfaces will require both an understanding of individual modalities and the fusion of information at various levels. The project is concerned, specifically, with the use of computer vision and speech processing as a means of interpreting and integrating information from spoken words, free hand gestures and gaze. It is also concerned with how to enable a human-computer dialogue with an interactive, multi-layered map in the context of a GIS and with how to enable map-mediated dialogue between human collaborators. The work will be bootstrapped through use of an existing test-bed that integrates gesture and speech for simple map queries. The research will investigate how characteristics of tasks, information, and users influence the nature of human-computer and computer-mediated human-human dialogue. In doing so, the research will determine how these factors influence the strategies required to achieve an effective easy-to-use interface that supports information access and productive work by individuals and groups.This research will advance the science of natural, multimodal human-computer interaction, while addressing, directly, the challenges of developing new knowledge through which advances in information technology can enable universal participation in geospatial information access and applications. Estimates suggest that as much as 80% of all digital data contain some form of georeferencing (coordinates, addresses, zip codes). As a result, GIS have become an important tool for managing and analyzing data across a range of science, policy analysis, resource management, business planning, and educational applications. In spite of the potential applications for GIS and the growing availability of geospatial data, GIS are underutilized because current systems are hard to use. There is a substantial gap between current GIS applications and the promise of GIS to facilitate public access to the large volume of geospatial information and to support public as well as expert input to important decisions affecting their communities and environments. Achieving the research goals outlined will, thus, enable more effective decision-making about risks to public safety and health and will support input to important public decisions by a wider range of participants.