Real-time Intersensory Discomfort Compensation

感觉错觉是创造可信的混合现实体验的关键。视觉错觉和通过动画图像创建虚拟世界的研究非常深入。为了将物理和虚拟更真实地融合在一起，多通道错觉在这种背景下非常强大。然而，对于许多人来说，这并不能很好地工作，因为他们在这些环境中感到不舒服。如果两个或多个感觉通道提供不同且相互冲突的信息，一种感觉通道可以用另一种覆盖另一种感觉通道的信息。到目前为止，还没有系统地理解多通道错觉如何创造一个令人信服的混合现实环境，从而避免不适。几个研究项目，包括我们之前的工作，已经证明了创造这种错觉的技术实现的可行性。在这个项目中，我们旨在系统地研究混合现实中多通道感觉错觉的基础。作为我们实验研究的一个关键方面，我们使用了混合现实中的不适现象。如果多感官信息不连贯，人类就会对此做出反应。如果认知没有整合多个感官的信息，我们就会感到不适。晕动病和晕睡病是这种感觉和看到的运动不匹配的例子，没有整合到连贯的认知中。尽管如此，人们还是会因为这种感官错配的影响而感到恶心。我们的想法是，在人们感到不适之前，使用生理感应来检测这种错配的开始。如果这是可能的，我们可以纠正错配，并在混合现实中创造出持久而全面的错觉。这建立在一项研究的基础上，该研究调查了在什么条件下使用技术可以实现感觉间的整合。这有助于系统地建立概念模型，说明什么感觉信息组合可以创造什么感觉错觉。我们通过生理感知来扩展这种静态模型，以克服认知模型所包含的内部和人际差异。我们的愿景是研究新一代混合现实设备和应用的科学基础，这些设备和应用可以感觉到错觉即将打破，并对此进行对抗。如果我们能够实时测量交互系统的个人用户在什么时候不能再整合多感官信息，系统就可以调整多感官输出，以避免不适。这方面的一个例子是一旦检测到打破错觉的开始就对视觉场景进行调整。如果可能，这可以使磁共振技术适用于更广泛的人和更广泛的应用，从而通过实时感觉间不适补偿创建一种新的、高效和有意义的交互范例。

Sensory illusions are key to creating believable mixed-reality experiences. Visual illusions and the creation of virtual worlds through animated images are well-researched. To advance towards a more realistic blending of the physical and the virtual, multimodal illusions are very powerful in this context. However, for many people that is not working well, as they feel discomfort in these settings. If two or more sensory modalities provide different and conflicting information, one sensory modality can overwrite the information sensed with another one. So far, there is no systematic understanding of how multimodal illusions can create a convincing mixed-reality environment that avoids discomfort. Several research projects, including our previous work, have demonstrated the feasibility of creating technical implementations of such illusions. In this project, we aim to systematically research the foundation of multimodal sensory illusions in mixed reality. As a key aspect of our experimental research, we use the phenomenon of discomfort in mixed reality. If the multisensory information is incoherent, humans react to this. If cognition does not integrate the information of multiple senses, we feel discomfort. Motion sickness and cybersickness are examples where this mismatch of felt and seen motion has not been integrated into coherent cognition. Still, instead, people feel sick as the effect of that sensory mismatch. Our idea is to use physiological sensing to detect the onset of such a mismatch before people feel discomfort. If this is possible, we could correct the mismatch and create a lasting and comprehensive illusion in mixed reality. This builds on research that has investigated under what conditions intersensory integration can be realized using technology. This helps to systematically build conceptual models of what sensory information combination can create what sensory illusion. We extend such static models by physiological sensing to overcome intra- and interpersonal differences cognitive models contain. The vision is to research the scientific foundation for a new generation of mixed-reality devices and applications, that can sense that the illusion is about to break, and counter this. If we can measure in real-time at what moment an individual user of an interactive system cannot integrate multisensory information anymore, the system could adapt the multisensory output to avoid discomfort. An example of this is the adaptation of the visual scene once the onset of the breaking illusion is detected. If possible, this could make MR technologies applicable to a much wider range of people and for a wider range of applications and, thus, create a novel efficient, and meaningful interaction paradigm through Real-time Intersensory Discomfort Compensation.