Collaborative Research: EAGER: Deep Learning-based Multimodal Analysis of Sleep

睡眠是动物和人类最基本的行为之一，了解群体睡眠将为神经科学、社会行为和相互作用提供关键的见解。为了克服单一模式动物行为平台的局限性，该项目将开发一种多模式机器学习方法，以同时监测和处理脑电(EEG)数据和动物行为数据，以系统地研究群体行为，特别是睡眠，并对动物的社会运动/行为进行注释。该项目的成果可能会为机械探索提供一个基于深度学习的强大工具包，以理解复杂的动物行为和脑电活动模式。这个项目的子问题将被开发成课程材料，并将成为本科学生的顶峰项目或指导研究。该项目将通过一个能够提取和聚合最相关信息的多模式机器学习框架，处理来自多个数据来源的多个实体的多种数据模式和小组活动。为了学习和处理长视频和脑电信号数据，将开发一部语义层面的动作词典，这对目前最先进的自我注意变压器模型来说是一个重大挑战。此外，为了纳入群体交互，将开发用于对话建模的转换器模型。拟议的同时进行的脑电和行为研究将提供群体睡眠的生物学基础，从而深入了解大脑电信号和行为输出-大脑如何组织其信号单元来产生行为。这一奖项反映了NSF的法定使命，并通过使用基金会的智力优势和更广泛的影响审查标准进行评估，被认为值得支持。

Sleep is one of the most fundamental behaviors for animals and humans, and understanding group sleep will provide key insights into neuroscience and social behavior and interactions. To overcome limitations that stem from single modality animal behavior platforms, the project will develop a multimodal machine learning method to simultaneously monitor and process Electroencephalogram (EEG) data and animal behavior data to systematically study group behavior, especially sleep, and to annotate animal social movements/behavior. The outcomes from the project will potentially provide a powerful toolkit based on deep learning to make sense of complex animal behavior and EEG activity pattern for mechanistic exploration. Subproblems from this project will be developed into course materials and will be capstone projects or directed study for undergraduate students.The project will process multiple data modalities and group activities involving multiple entities from multiple data sources through a multi-modal machine learning framework enabling the extraction and aggregation of the most pertinent information. A “dictionary” of movements at the semantic level will be developed for learning and processing of long video and EEG data, which is a significant challenge for current state-of-the-art self-attention transformer models. Additionally, to incorporate group interactions, transformer models for dialogue modeling will be developed. The proposed simultaneous EEG and behavior study will provide biological underpinnings of group sleep, leading to insights into brain electrical signaling and behavioral outputs - how the brain marshals its signaling units to generate behaviors.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.