Developing Effective Immersive Environments: Multimodal Interactions, and User Interface Customization

点击翻译按钮获取中文摘要

Accurately simulating the sense of touch is complex, typically requiring high-end haptic devices that are cost-prohibitive for most training applications that rely on technical skills. Although consumer-level haptic devices are becoming more widespread given their reasonable cost, they are generally restrictive given their inherent lower level of fidelity, and lower range of motion required to realistically simulate many tasks. In the real-world our senses are constantly exposed to stimuli from multiple sensory modalities (visual, auditory, vestibular, olfactory, and haptic), and we are able to integrate/process this multisensory information and acquire knowledge of multisensory objects. The interactions between the auditory and visual modalities has long been established yet, very little prior work has considered multimodal interactions with the other senses, particularly the sense of touch. Within the scope of the proposed research program, a methodical investigation of multimodal interactions (with an emphasis on sound, and visuals on haptic fidelity and task performance), within virtual learning environments will be made through a series of human-based experiments. In the first phase, the experiments will involve simple drilling-based tasks (with a custom 3D printed haptic-based drill), where single parameters (e.g., auditory fidelity) will be varied and the resulting effect measured. In the second and third phases, a lab fire safety serious game will be modified to incorporate a 3D printed haptic-based fire extinguisher that can be used to extinguish a fire and provide a more realistic scenario to examine multimodal interactions. Similar to phase 1, parameters will be varied and the resulting effects will be measured. In both phases, qualitative (e.g., questionnaires), and quantitative (e.g., physiological), measures will be made. Based on the collective results, a set of best practice standards will be devised to help developers of virtual learning environments develop more effective training tools using low-end haptic devices and multimodal interactions. Finally, in the fourth phase, a novel method will be developed to allow various aspects of a virtual environment (e.g., auditory and visual fidelity), to be tailored specifically to the user via a brief calibration process that the user will complete prior to using the virtual environment. Collectively, the proposed research program will advance our understanding of multi-modal virtual learning environments and will result in more effective multimodal virtual simulations and serious games, ultimately leading to a better trained workforce. Furthermore, students carrying out the proposed research will work with state-of-the-art equipment, on cutting-edge research within an interdisciplinary environment allowing them to acquire and develop various skills (leadership, technical, communication, etc.), highly sought in academia and industry alike.