A hands-free interactive system for manipulating object-oriented information within networked virtual environments

虚拟现实（VR）旨在通过视觉和触觉等多感官方式进行实时模拟和交互，在包括石油和天然气勘探在内的许多领域都有应用。然而，由于与直观的用户交互相关的瓶颈问题，VR应用的适应一直很慢。当VR应用程序在计算机网络上运行以进行多用户交互时，这个问题变得更加成问题。我们建议在网络虚拟环境中创建一个操作面向对象信息的免提交互系统。开发自给自足的桌面虚拟现实，实现互联互通，形成网络化虚拟环境。每个桌面虚拟现实将生成三维立体的油藏模拟显示-计算机建模集团有限公司（CMG）的软件产品。支撑桌面VR的将是一个免提交互系统。这个互动系统将利用基于摄像头的手势识别（提取用户与3D显示器对象交互的意图）和触觉（通过向用户的手提供力反馈来模拟触觉）。整合手势识别和触觉的框架将采用人类用于执行视觉感知和视觉引导动作的神经机制。交互系统将允许多个用户在网络虚拟环境中协作他们的个人交互（通过每个桌面VR）。该项目的新颖之处在于将手势识别与触觉技术相结合，实现用户交互；使用新的3D测距相机进行手势识别；以及神经机制在解决工程问题中的应用。我们之前的工作证实，利用现有的技术和知识，创建免提交互系统是可以实现的。作为油藏模拟的一种新型工具，该交互式系统将推动加拿大有效的石油相关服务和经济发展。它的好处是使CMG能够领导储层模拟的利基市场。它的影响将超越与石油相关的服务，作为连接和增强其他应用创新vr技术的桥梁。

Aiming at real-time simulation and interaction via multi-sensory modalities such as vision and touch, virtual reality (VR) finds many applications including oil and gas exploration. However, adaptation of VR applications has been slow, due to a bottle-neck issue related to intuitive user interaction. This issue becomes more problematic when VR applications run across a computer network for multi-user interaction. We propose to create a hands-free interactive system for manipulating object-oriented information within networked virtual environments. We will develop self-contained desktop VRs and interconnect them to form a networked virtual environment. Each desktop VR will generate 3D stereoscopic displays of the Reservoir Simulation - a software product from the Computer Modelling Group Ltd (CMG). Underpinning the desktop VR will be a hands-free interactive system. This interactive system will utilize camera-based gesture recognition (extracting the user intention of interacting with objects of the 3D displays) and haptics (simulating the sense of touch by providing force feedback to the user's hand). The framework of integrating both gesture recognition and haptics will employ the neural mechanisms that humans use to perform visual perception and visually guided actions. The interactive system will allow multiple users to collaborate their individual interaction (via each desktop VR) across the networked virtual environment. The novelty of the project lies in the integration of gesture recognition and haptics for user interaction; the use of new 3D range cameras for gesture recognition; and the application of the neural mechanisms to address engineering issues. Our previous work confirms that it is achievable to create the hands-free interactive system using current technologies and knowledge. As a novel tool for reservoir simulation, the hands-free interactive system will propel effective oil-related services and economic development for Canada. Its benefit will be allowing the CMG to lead a niche market of reservoir simulation. Its impact will go beyond the oil-related services, serving as a bridge to connect and enhance innovative VR-based technologies for other applications.