Collaborative Research: EAGER: Deep Learning-based Multimodal Analysis of Sleep

睡眠是动物和人类最基本的行为之一，理解群体睡眠将为神经科学、社会行为和互动提供关键的见解。为了克服单模态动物行为平台的局限性，该项目将开发一种多模态机器学习方法，同时监测和处理脑电图（EEG）数据和动物行为数据，系统地研究群体行为，特别是睡眠，并注释动物的社会运动/行为。该项目的结果可能会提供一个基于深度学习的强大工具包，以理解复杂的动物行为和脑电图活动模式，以进行机械探索。本计画的子问题将会发展成课程教材，并会成为本科学生的顶点计画或指导研究。该项目将通过多模态机器学习框架处理涉及多个数据源的多个实体的多种数据模式和组活动，从而提取和聚合最相关的信息。语义层面的运动“字典”将用于学习和处理长视频和脑电图数据，这是目前最先进的自注意变压器模型的一个重大挑战。此外，为了结合群体互动，将开发用于对话建模的变压器模型。同时提出的脑电图和行为研究将为群体睡眠提供生物学基础，从而深入了解大脑电信号和行为输出——大脑如何组织其信号单元来产生行为。该奖项反映了美国国家科学基金会的法定使命，并通过使用基金会的知识价值和更广泛的影响审查标准进行评估，被认为值得支持。

Sleep is one of the most fundamental behaviors for animals and humans, and understanding group sleep will provide key insights into neuroscience and social behavior and interactions. To overcome limitations stemmed from single modality animal behavior platforms, the project will develop a multimodal machine learning method to simultaneously monitor and process the Electroencephalogram (EEG) data and animal behavior data to systematically study group behavior, especially sleep, and to annotate animal social movements/behavior. The outcomes from the project will potentially provide a powerful toolkit based on deep learning to make sense of complex animal behavior and EEG activity pattern for mechanistic exploration. Subproblems from this project will be developed into course materials and will be capstone projects or directed study for undergraduate students.The project will process multiple data modalities and group activities involving multiple entities from multiple data sources through a multi-modal machine learning framework enabling the extraction and aggregation of the most pertinent information. A “dictionary” of movements at the semantic level will be developed for learning and processing of long video and EEG data, which is a significant challenge for current state-of-the-art self-attention transformer models. Additionally to incorporate group interactions, transformer models for dialogue modeling will be developed. The proposed simultaneous EEG and behavior study will provide biological underpinnings of group sleep, leading to insights into brain electrical signaling and behavioral outputs - how the brain marshals its signaling units to generate behaviors.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.