Real-time Intersensory Discomfort Compensation

感官错觉是创造可信的混合现实体验的关键。视觉错觉和通过动画图像创建虚拟世界是很好的研究。为了进一步实现物理和虚拟的更真实的融合，多模态错觉在这种情况下非常强大。然而，对于许多人来说，这并不奏效，因为他们在这些环境中感到不适。如果两个或多个感觉模态提供不同且冲突的信息，则一个感觉模态可以用另一个感觉模态覆盖所感测的信息。到目前为止，还没有系统的理解多模态错觉如何创造一个令人信服的混合现实环境，避免不适。几个研究项目，包括我们以前的工作，已经证明了创造这种错觉的技术实现的可行性。在这个项目中，我们的目标是系统地研究混合现实中的多通道感觉错觉的基础。作为我们实验研究的一个关键方面，我们使用混合现实中的不适现象。如果多感官信息是不连贯的，人类会对此做出反应。如果认知不能整合多种感官的信息，我们就会感到不适。晕动病和电脑病就是这种感觉和看到的运动不匹配没有被整合到连贯认知中的例子。然而，相反，人们会因为感觉不匹配而感到恶心。我们的想法是使用生理传感来检测这种不匹配的发生，然后人们才会感到不适。如果这是可能的，我们可以纠正这种不匹配，并在混合现实中创造一个持久而全面的幻觉。这建立在研究的基础上，研究了在什么条件下可以使用技术实现感觉间整合。这有助于系统地建立什么感觉信息组合可以创造什么感觉错觉的概念模型。我们通过生理感知来扩展这种静态模型，以克服认知模型中包含的内部和人际差异。我们的愿景是研究新一代混合现实设备和应用程序的科学基础，这些设备和应用程序可以感知到幻觉即将打破，并对此进行反击。如果我们能够实时测量交互系统的个体用户在什么时候不能再整合多感官信息，系统就可以调整多感官输出以避免不适。这方面的一个例子是，一旦检测到破坏性错觉的开始，就对视觉场景进行调整。如果可能的话，这可以使MR技术适用于更广泛的人群和更广泛的应用，从而通过实时感觉间不适补偿创造一种新的高效和有意义的交互模式。

Sensory illusions are key to creating believable mixed-reality experiences. Visual illusions and the creation of virtual worlds through animated images are well-researched. To advance towards a more realistic blending of the physical and the virtual, multimodal illusions are very powerful in this context. However, for many people that is not working well, as they feel discomfort in these settings. If two or more sensory modalities provide different and conflicting information, one sensory modality can overwrite the information sensed with another one. So far, there is no systematic understanding of how multimodal illusions can create a convincing mixed-reality environment that avoids discomfort. Several research projects, including our previous work, have demonstrated the feasibility of creating technical implementations of such illusions. In this project, we aim to systematically research the foundation of multimodal sensory illusions in mixed reality. As a key aspect of our experimental research, we use the phenomenon of discomfort in mixed reality. If the multisensory information is incoherent, humans react to this. If cognition does not integrate the information of multiple senses, we feel discomfort. Motion sickness and cybersickness are examples where this mismatch of felt and seen motion has not been integrated into coherent cognition. Still, instead, people feel sick as the effect of that sensory mismatch. Our idea is to use physiological sensing to detect the onset of such a mismatch before people feel discomfort. If this is possible, we could correct the mismatch and create a lasting and comprehensive illusion in mixed reality. This builds on research that has investigated under what conditions intersensory integration can be realized using technology. This helps to systematically build conceptual models of what sensory information combination can create what sensory illusion. We extend such static models by physiological sensing to overcome intra- and interpersonal differences cognitive models contain. The vision is to research the scientific foundation for a new generation of mixed-reality devices and applications, that can sense that the illusion is about to break, and counter this. If we can measure in real-time at what moment an individual user of an interactive system cannot integrate multisensory information anymore, the system could adapt the multisensory output to avoid discomfort. An example of this is the adaptation of the visual scene once the onset of the breaking illusion is detected. If possible, this could make MR technologies applicable to a much wider range of people and for a wider range of applications and, thus, create a novel efficient, and meaningful interaction paradigm through Real-time Intersensory Discomfort Compensation.