Collaborative Research: EAGER: Deep Learning-based Multimodal Analysis of Sleep

睡眠是动物和人类最基本的行为之一，了解群体睡眠将为神经科学和社会行为及相互作用提供关键见解。为了克服单一模态动物行为平台的局限性，该项目将开发一种多模态机器学习方法，同时监测和处理脑电图（EEG）数据和动物行为数据，以系统地研究群体行为，特别是睡眠，并注释动物的社会运动/行为。该项目的成果可能会提供一个基于深度学习的强大工具包，以了解复杂的动物行为和EEG活动模式，从而进行机械探索。该项目的子问题将被开发成课程材料，并将成为本科生的顶级项目或指导性研究。该项目将通过多模态机器学习框架处理涉及多个数据源的多个实体的多个数据模态和组活动，从而能够提取和聚合最相关的信息。将开发语义级别的运动“字典”，用于学习和处理长视频和EEG数据，这对当前最先进的自我注意力Transformer模型是一个重大挑战。此外，将开发用于对话建模的Transformer模型，以纳入组交互。拟议的同时进行的EEG和行为研究将提供群体睡眠的生物学基础，从而深入了解大脑电信号和行为输出--大脑如何排列其信号单元以产生行为。该奖项反映了NSF的法定使命，并且通过使用基金会的智力价值和更广泛的影响审查标准进行评估，被认为值得支持。

Sleep is one of the most fundamental behaviors for animals and humans, and understanding group sleep will provide key insights into neuroscience and social behavior and interactions. To overcome limitations stemmed from single modality animal behavior platforms, the project will develop a multimodal machine learning method to simultaneously monitor and process the Electroencephalogram (EEG) data and animal behavior data to systematically study group behavior, especially sleep, and to annotate animal social movements/behavior. The outcomes from the project will potentially provide a powerful toolkit based on deep learning to make sense of complex animal behavior and EEG activity pattern for mechanistic exploration. Subproblems from this project will be developed into course materials and will be capstone projects or directed study for undergraduate students.The project will process multiple data modalities and group activities involving multiple entities from multiple data sources through a multi-modal machine learning framework enabling the extraction and aggregation of the most pertinent information. A “dictionary” of movements at the semantic level will be developed for learning and processing of long video and EEG data, which is a significant challenge for current state-of-the-art self-attention transformer models. Additionally to incorporate group interactions, transformer models for dialogue modeling will be developed. The proposed simultaneous EEG and behavior study will provide biological underpinnings of group sleep, leading to insights into brain electrical signaling and behavioral outputs - how the brain marshals its signaling units to generate behaviors.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.