Developing Effective Immersive Environments: Multimodal Interactions, and User Interface Customization

准确模拟触觉是复杂的，通常需要高端触觉设备，对于依赖技术技能的大多数培训应用来说，这些设备成本高昂。尽管消费级触觉设备由于其合理的成本而变得越来越普遍，但是由于其固有的较低保真度水平和逼真地模拟许多任务所需的较低运动范围，它们通常是限制性的。在现实世界中，我们的感官不断暴露于来自多种感官形式（视觉，听觉，前庭，嗅觉和触觉）的刺激，我们能够整合/处理这些多感官信息并获得多感官对象的知识。听觉和视觉模态之间的相互作用早已建立，但之前的工作很少考虑与其他感官（特别是触觉）的多模态相互作用。 在拟议的研究计划的范围内，将通过一系列以人为本的实验，在虚拟学习环境中进行多模态交互（重点是声音和视觉触觉保真度和任务性能）的系统调查。在第一阶段，实验将涉及简单的基于钻孔的任务（使用定制的3D打印的基于触觉的钻头），其中单个参数（例如，听觉保真度）将被改变并且测量所产生的效果。在第二阶段和第三阶段，将修改实验室消防安全严肃游戏，以纳入3D打印的基于触觉的灭火器，该灭火器可用于灭火，并提供更逼真的场景来检查多模态交互。与第一阶段类似，将改变参数，并测量产生的效果。在这两个阶段，定性（例如，问卷调查），和定量（例如，生理学），将采取措施。根据集体的结果，将设计一套最佳实践标准，以帮助虚拟学习环境的开发人员使用低端触觉设备和多模式交互开发更有效的培训工具。最后，在第四阶段，将开发一种新的方法，以允许虚拟环境的各个方面（例如，听觉和视觉保真度），以通过用户在使用虚拟环境之前完成的简短校准过程来专门为用户定制。 总的来说，拟议的研究计划将促进我们对多模态虚拟学习环境的理解，并将导致更有效的多模态虚拟模拟和严肃的游戏，最终导致更好的训练有素的劳动力。此外，进行拟议研究的学生将使用最先进的设备，在跨学科环境中进行尖端研究，使他们能够获得和发展各种技能（领导力，技术，沟通等），在学术界和工业界都备受追捧。

Accurately simulating the sense of touch is complex, typically requiring high-end haptic devices that are cost-prohibitive for most training applications that rely on technical skills. Although consumer-level haptic devices are becoming more widespread given their reasonable cost, they are generally restrictive given their inherent lower level of fidelity, and lower range of motion required to realistically simulate many tasks. In the real-world our senses are constantly exposed to stimuli from multiple sensory modalities (visual, auditory, vestibular, olfactory, and haptic), and we are able to integrate/process this multisensory information and acquire knowledge of multisensory objects. The interactions between the auditory and visual modalities has long been established yet, very little prior work has considered multimodal interactions with the other senses, particularly the sense of touch. Within the scope of the proposed research program, a methodical investigation of multimodal interactions (with an emphasis on sound, and visuals on haptic fidelity and task performance), within virtual learning environments will be made through a series of human-based experiments. In the first phase, the experiments will involve simple drilling-based tasks (with a custom 3D printed haptic-based drill), where single parameters (e.g., auditory fidelity) will be varied and the resulting effect measured. In the second and third phases, a lab fire safety serious game will be modified to incorporate a 3D printed haptic-based fire extinguisher that can be used to extinguish a fire and provide a more realistic scenario to examine multimodal interactions. Similar to phase 1, parameters will be varied and the resulting effects will be measured. In both phases, qualitative (e.g., questionnaires), and quantitative (e.g., physiological), measures will be made. Based on the collective results, a set of best practice standards will be devised to help developers of virtual learning environments develop more effective training tools using low-end haptic devices and multimodal interactions. Finally, in the fourth phase, a novel method will be developed to allow various aspects of a virtual environment (e.g., auditory and visual fidelity), to be tailored specifically to the user via a brief calibration process that the user will complete prior to using the virtual environment. Collectively, the proposed research program will advance our understanding of multi-modal virtual learning environments and will result in more effective multimodal virtual simulations and serious games, ultimately leading to a better trained workforce. Furthermore, students carrying out the proposed research will work with state-of-the-art equipment, on cutting-edge research within an interdisciplinary environment allowing them to acquire and develop various skills (leadership, technical, communication, etc.), highly sought in academia and industry alike.