CAREER: Multimodal Brain and Body Music Interfaces to Promote Entrainment, Connection, and Creative Science Education

该奖项的部分资金来自《2021年美国救援计划法案》(公法117-2)。夹带是一个过程，在这个过程中，人们的大脑和身体的自然节奏通过音乐等刺激同步，这可能会产生联系和幸福的感觉。该项目通过构建多模式信号映射接口来解决夹带问题，该接口通过从大脑和身体节奏中提取音乐来调节人际联系。研究人员将整合传感器硬件和信号处理软件，以流式传输实时大脑和身体数据，执行计算以提取信号特征，并利用此来驱动声音合成。一系列的音乐认知和听实验研究了生理、行为和情感的夹带现象，这些现象可能是一系列多模式大脑音乐接口的结果。在与医生协商后开发的一个用例研究，使用多模式夹带界面将物理上相隔很远的母亲和婴儿联系起来。母亲和婴儿听到的音乐源自彼此的心跳和呼吸。这项研究调查了他们在身体节律中产生的缠绕，并绘制了虚拟连接环境对健康和幸福的影响。对于研究人员、医生和照顾者来说，多模式脑音乐界面有可能扩大我们对音乐对大脑和身体有益影响的科学理解，这可能会为成人、儿童和婴儿带来新的健康和福祉干预。该项目将产生一个开放源码工具包，其中包括可获得的技术和STEM学习模块，以激励教育工作者和学生开发项目，促进我们对大脑和身体信号的理解。这些学习模块将被整合到一次暑期研究体验中--涉及高中生和他们的老师--在这种体验中，真正的学习鼓励学生通过他们对音乐的自然兴趣来进行科学方法的训练。这个项目开发和评估了一个与新的多模式信号映射技术的接口，该接口将神经生理信号(例如，脑电、心电、EDA、呼吸)转换成音乐声音，以促进个人和计算机之间的生物、行为和情感同步，方法是：(1)设计可发声技术，执行实时信号处理和算法音乐生成，将生理信号转换为音乐；(2)通过设计新的有节奏的听觉神经生理发声刺激和测量人体如何反应来研究支配听觉神经刺激和生理夹带的神经心理学机制；以及(3)设计和评估一个用例，其中包括使用彼此的生理数据为婴儿和他们的母亲共同生成音乐。定量数据将解决生理同步问题，视频协议分析将解决行为同步问题，定性数据将解决体验问题。这些研究活动将有助于实现一个总体目标，即发现如何使用计算将音乐和生理学配对，从而在以人为中心的计算中发挥重要的信息渠道的作用。这个渠道的一个预期用途是通过夹带促进人与人之间的联系和福祉。这一奖项反映了NSF的法定使命，并通过使用基金会的智力优势和更广泛的影响审查标准进行评估，被认为值得支持。

This award is funded in part under the American Rescue Plan Act of 2021 (Public Law 117-2). Entrainment is a process in which people’s natural brain and body rhythms synchronize, through stimuli such as music, which may create feelings of connection and well-being. This project addresses entrainment by building multimodal signal mapping interfaces that mediate interpersonal connections by deriving music from brain and body rhythms. The investigator will integrate sensor hardware and signal processing software to stream live brain and body data, perform calculations to extract signal characteristics, and use this to drive sound synthesis. A series of music cognition and listening experiments study physiological, behavioral, and affective entrainment phenomena, which are expected to result, from a series of multimodal brain music interfaces. A use-case study, developed in consultation with doctors, connects mothers and infants, physically separated by distance, using the multimodal entrainment interface. Mother and infant hear music derived from each other’s heartbeats and breathing. This study investigates the entrainment created in their body rhythms, and maps health and well-being effects of the virtual connection environment. For researchers, doctors, and caretakers, multimodal brain music interfaces have the potential to expand our scientific understanding of music’s beneficial effects on the brain and body, which may lead to new health and well-being interventions for adults, children, and infants. This project will result in an open-source tool kit of accessible technologies and STEM learning modules to inspire educators and students to develop projects that further our understanding of brain and body signals. These learning modules will be integrated into a summer research experience--involving high school students and their teachers--in which authentic learning encourages students’ training in the scientific method through their natural interest in music. This project develops and evaluates an interface with new multimodal signal mapping technologies that translate neurophysiological signals (e.g., EEG, ECG, EDA, respiration) into musical sound to promote biological, behavioral, and affective synchrony between individuals and computers by: (1) engineering sonification techniques that perform real-time signal processing and algorithmic music generation for transforming physiological signals into music; (2) investigating the neuropsychological mechanisms that govern auditory neurostimulation and physiological entrainment by designing new rhythmic auditory neurophysiological sonification stimuli and measuring how the human body responds; and (3) designing and evaluating a use case that involves co-generating music for infants and their mothers with each other’s physiological data. Quantitative data will address synchronies in physiology, protocol analysis of video will address behavioral synchronies, and qualitative data will address experiences. These research activities will contribute to an overarching goal of discovering how using computing to pair music and physiology can function as a significant information channel in human-centered computing. One expected use of this channel is to promote human connection and well-being through entrainment.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.