Immersive Neuroscience: Bringing the Study of the Human Brain and Behavior Closer to Real Life

我的研究的首要目标是了解人类大脑如何使用视觉来感知世界和执行行动。这一授权期的短期目标是培养一种新的方法，我称之为沉浸式神经科学，使用新技术来研究真实世界中的大脑和行为，以及引人注目的模拟。在我过去的三次NSERC发现奖中，我开发了使用人类神经成像(特别是功能磁共振成像，fMRI)来研究真实动作和真实对象的技术。这项研究表明，真实的刺激和真实的行动比模拟(如哑剧中的动作和图像)更吸引人--无论是在行为上还是在神经上都是如此，这一现象我们称之为“真实世界的优势”。虽然结果具有启发性，但使用功能磁共振成像来研究真实情况是非常具有挑战性的，也有局限性。新兴技术为沉浸式神经科学提供了新的发展途径。首先，3D显示和虚拟/增强现实技术能够呈现引人入胜的真实世界场景，使认知神经科学超越了对静态2D图像的严重依赖。其次，新的光学神经成像技术(功能性近红外光谱、fNIRS和核心流等高分辨率方法)使在真实世界中直接测量大脑皮质反应成为可能。我的研究计划将利用这些技术开创新的方法，使认知神经科学研究更接近现实世界。这笔赠款将集中在以下三个方面的研究。首先，我将研究人脑如何处理复杂的三维视觉刺激，如人脸、场景和物体。其次，我将研究与图像或视频相比，现实主义的哪些方面(特别是3D视觉和可操作性)改变了人们对真实对象和真实人物的行为和大脑反应。第三，我将研究在现实任务中大脑是如何运作的，在现实任务中，参与者主动控制他们的环境，并获得持续的反馈。沉浸式神经科学将为下一代神经科学提供丰富的机会，通过推动认知神经科学的现实主义，同时保持对精心设计的实验的足够控制，从而更接近日常生活。这种方法提供了对现实主义至关重要的因素进行创新发现的潜力。此外，它提高了虚拟现实(例如，培训、图像引导手术)和光学神经成像(例如，脑机接口)应用的成功可能性。

The overarching goal of my research is to understand how the human brain uses vision to perceive the world and perform actions. The short-term goal for this granting period is to cultivate a new approach that I call Immersive Neuroscience to use novel technologies to study the brain and behavior in the real world and compelling simulations. Over my past three NSERC Discovery grants, I have developed techniques to study real actions and real objects using human neuroimaging (specifically functional magnetic resonance imaging, fMRI). This research suggests that real stimuli and real actions are more engaging -- both behaviorally and neurally -- than simulations (such as pantomimed actions and images), a phenomenon we call the "real-world advantage". While the results have been suggestive, using fMRI to study real situations is very challenging and has limitations. Emerging technologies are providing new ways forward for Immersive Neuroscience. First, 3D displays and virtual/augmented reality technologies enable the presentation of compelling real-world scenes, enabling cognitive neuroscience to move beyond heavy reliance on static 2D images. Second, new optical neuroimaging technologies (functional near infrared spectroscopy, fNIRS, and high-resolution approaches like the Kernel Flow) enable the measurement of cortical brain responses directly in the real world. My research program will pioneer new approaches with these technologies to bring cognitive neuroscience research closer to the real world. This grant will focus on the following three lines of research. First, I will examine how the human brain processes complex three-dimensional visual stimuli like faces, scenes and objects. Second, I will examine which aspects of realism (particularly 3D vision and actability) change behavior and brain responses to real objects and real people compared to images or videos. Third, I will examine how the brain functions during realistic tasks in which participants actively control their environments and get ongoing feedback. Immersive Neuroscience will provide rich opportunities for next-generation neuroscience to move closer to everyday life by pushing the realism of cognitive neuroscience while maintaining sufficient control for carefully designed experiments. This approach offers the potential to make innovative discoveries about the factors crucial to realism. Moreover, it enhances the likelihood of success for applications of virtual reality (e.g., training, image-guided surgery) and optical neuroimaging (e.g., brain-computer interfaces).