Developing Effective Immersive Environments: Multimodal Interactions, and User Interface Customization

准确地模拟触觉是复杂的，通常需要高端触觉设备，对于大多数依赖技术技能的培训应用来说，这些设备的成本过高。尽管消费者级别的触觉设备正变得越来越普遍，因为它们的价格合理，但它们通常受到限制，因为它们固有的较低水平的保真度，以及逼真地模拟许多任务所需的运动范围较小。在现实世界中，我们的感官不断地受到来自多种感官形式(视觉、听觉、前庭、嗅觉和触觉)的刺激，我们能够整合/处理这些多感官信息，并获得关于多感官对象的知识。听觉和视觉形式之间的相互作用早已确定，但以前的工作很少考虑与其他感官，特别是触觉的多模式相互作用。在拟议的研究计划范围内，将通过一系列以人为基础的实验，对虚拟学习环境中的多模式互动(重点是声音和视觉，以及触觉保真度和任务表现)进行有条不紊的调查。在第一阶段，实验将涉及简单的基于钻探的任务(使用定制的3D打印触觉钻探)，其中单个参数(例如，听觉保真度)将发生变化，并测量结果。在第二和第三阶段，将对实验室消防安全严肃游戏进行修改，以纳入3D打印触觉灭火器，该灭火器可用于扑灭火灾，并提供更逼真的场景来检查多模式互动。与第一阶段类似，参数将会变化，由此产生的影响将被测量。在定性(例如，问卷)和定量(例如，生理)两个阶段，都将采取措施。根据收集的结果，将制定一套最佳实践标准，以帮助虚拟学习环境的开发人员使用低端触觉设备和多模式互动开发更有效的培训工具。最后，在第四阶段，将开发一种新的方法，以允许通过用户将在使用虚拟环境之前完成的简短校准过程来为用户专门定制虚拟环境的各个方面(例如，听觉和视觉保真度)。总的来说，拟议的研究计划将促进我们对多模式虚拟学习环境的理解，并将导致更有效的多模式虚拟模拟和严肃的游戏，最终导致更好的培训劳动力。此外，进行拟议研究的学生将使用最先进的设备，在跨学科环境中进行尖端研究，使他们能够获得和发展各种技能(领导力、技术、沟通等)，这些技能在学术界和工业界都很受欢迎。

Accurately simulating the sense of touch is complex, typically requiring high-end haptic devices that are cost-prohibitive for most training applications that rely on technical skills. Although consumer-level haptic devices are becoming more widespread given their reasonable cost, they are generally restrictive given their inherent lower level of fidelity, and lower range of motion required to realistically simulate many tasks. In the real-world our senses are constantly exposed to stimuli from multiple sensory modalities (visual, auditory, vestibular, olfactory, and haptic), and we are able to integrate/process this multisensory information and acquire knowledge of multisensory objects. The interactions between the auditory and visual modalities has long been established yet, very little prior work has considered multimodal interactions with the other senses, particularly the sense of touch. Within the scope of the proposed research program, a methodical investigation of multimodal interactions (with an emphasis on sound, and visuals on haptic fidelity and task performance), within virtual learning environments will be made through a series of human-based experiments. In the first phase, the experiments will involve simple drilling-based tasks (with a custom 3D printed haptic-based drill), where single parameters (e.g., auditory fidelity) will be varied and the resulting effect measured. In the second and third phases, a lab fire safety serious game will be modified to incorporate a 3D printed haptic-based fire extinguisher that can be used to extinguish a fire and provide a more realistic scenario to examine multimodal interactions. Similar to phase 1, parameters will be varied and the resulting effects will be measured. In both phases, qualitative (e.g., questionnaires), and quantitative (e.g., physiological), measures will be made. Based on the collective results, a set of best practice standards will be devised to help developers of virtual learning environments develop more effective training tools using low-end haptic devices and multimodal interactions. Finally, in the fourth phase, a novel method will be developed to allow various aspects of a virtual environment (e.g., auditory and visual fidelity), to be tailored specifically to the user via a brief calibration process that the user will complete prior to using the virtual environment. Collectively, the proposed research program will advance our understanding of multi-modal virtual learning environments and will result in more effective multimodal virtual simulations and serious games, ultimately leading to a better trained workforce. Furthermore, students carrying out the proposed research will work with state-of-the-art equipment, on cutting-edge research within an interdisciplinary environment allowing them to acquire and develop various skills (leadership, technical, communication, etc.), highly sought in academia and industry alike.