Plausible multisensory modeling of 3D deformable objects for interactive applications

在市场上推出3D支持的技术，如3D显示器、3D眼镜以及与虚拟环境交互的各种手段(手套、触觉反馈设备等)给研究界带来了越来越大的压力，要求他们开发3D内容的启用技术。为了充分发挥其潜力，交互环境需要超越软件生成对象的建模，引入基于真实测量的3D对象的合理表示。现实世界中的对象所表现出的丰富的交互行为不仅取决于对象的外观，还取决于它们的属性：它们有多硬，它们的表面在触摸时的感觉和声音如何，或者它们在接触时如何变形。基于真实测量的对象复杂行为的交互模型的开发没有跟随娱乐和计算机游戏行业中实现的视觉和音频真实感的快速发展。面对这一挑战，该研究计划的长期目标是通过基于真实多感觉测量的可信3D对象模型及其集成、表示和解释，为需要更高真实感的应用开发增强的传感和建模技术。从这一长期目标涉及的众多方面来看，这一最初的研究计划侧重于三个主要方面：(1)获取视觉、听觉和触觉测量并将其整合到逼真、连贯的3D模型中；(2)渲染此类对象的复杂行为；以及(3)在特定应用的背景下整合和评估多感官对象模型。更具体地说，人类视觉和触觉感知方法、使用各种传感器的触觉和视觉采样策略、视觉算法和计算智能技术将被调整和改进，以用于实际测量。它们将有利地组合在一起，以基于多感官视觉、触觉和音频数据来形成新的对象模型。原创的计算智能方法将被提出，以允许有效地使用关于对象行为的各种信息源，并在交互环境中无缝地合并多感官模型。在初始阶段，将实施一个专用界面，用于可视化和评估所开发的模型。接下来，这些模型将在洞穴自动虚拟环境(CAVE)中进行验证，在那里可以立即察觉到微小的缺陷。模型最终将被集成到两个实际应用中，一个用于恐惧症治疗，另一个用于基于模型的可变形对象的安全操作。拟议的计划将提供所需的理论和实践知识，以建立下一代互动应用程序，这些应用程序将依赖于从实际测量获得的可信的3D内容，而不是硬编码的近似计算机生成的图形模型。这一研究领域的进展将对需要更高精度和更好真实感的交互式应用程序产生重大影响。除上述两种应用外，这些应用还包括用于培训的交互式环境、用于对精度敏感的工业CAD流程的交互式虚拟样机、智能制造以及机器人装配、遥控机器人、医疗和辅助机器人等灵巧操纵应用。这项研究工作将支持对具有尖端方法和技术的高素质专家的培训，这些专家对加拿大强大的多媒体、游戏和虚拟现实产业有着巨大的需求。

The introduction on the market of 3D enabled technologies, such as 3D displays, 3D glasses and of various means for interaction with virtual environments (gloves, tactile feedback equipment, etc.) places an increasing pressure on the research community to develop enabling technologies for 3D contents. To fully reach their potential, interactive environments need to go beyond the modeling of software generated objects by introducing plausible representations of 3D objects based on real measurements. The rich interaction behavior exhibited by real-world objects depends not only on how objects look, but also on their properties: how stiff they are, how their surface feels and sounds when touched, or how they deform on contact. The development of interactive models for the complex behavior of objects based on real measurements has not followed the rapid progress of visual and audio realism achieved in the entertainment and computer gaming industry. Rising up to this challenge, the long-term goal of this research program is to develop enhanced sensing and modeling techniques for applications that require higher realism by means of plausible 3D object models based on real multisensory measurements, along with their integration, representation and interpretation. From the multitude of aspects involved in this long-term objective, this original research program focuses on three major aspects: (1) the acquisition and integration of visual, audio and tactile measurements into realistic, coherent 3D models; (2) the rendering of the complex behavior of such objects; and (3) the integration and evaluation of multisensory object models in the context of specific applications. More specifically, human visual and tactile sensing approaches, tactile and visual sampling strategies using various sensors, vision algorithms and computational intelligence techniques will be adapted and refined for use on real measurements. They will be advantageously combined for the formulation of novel object models based on multisensory visual, tactile and audio data. Original computational intelligence methods will be proposed to allow for an efficient use of various sources of information on the object behavior and for the seamless merge of multisensory models in interactive environments. In an initial stage, a dedicated interface will be implemented for the visualization and evaluation of the developed models. Next, the models will be validated in a Cave Automatic Virtual Environment (CAVE) where small imperfections become immediately perceptible. Models will finally be integrated into two practical applications, one for phobia treatment and the other one for safe model-based object manipulation of deformable objects. The proposed program will contribute the theoretical and practical knowledge required to build the next generation of interactive applications that will rely on plausible 3D contents acquired from real measurements, instead of hard-coded, approximate computer-generated graphics models. Advancements in this area of research will have a significant impact on interactive applications that require higher accuracy and improved realism. These include, beyond the two applications above, interactive environments for training, interactive virtual prototyping for precision-sensitive industrial CAD-based processes, smart manufacturing as well as dexterous manipulation applications such as robotic assembly, tele-robotics, medical and assistive robotics. This research work will support the training of highly qualified specialists with cutting edge approaches and technologies, who are in demand for the strong multimedia, gaming, and virtual reality industry in Canada.