NCS-FO:Tracking social behavior and its neural properties in a smart aviary

NCS-FO：跟踪智能鸟舍中的社会行为及其神经特性

• 批准号: 2124355
• 负责人: Marc Schmidt
• 金额: $ 99.36万
• 依托单位: University of Pennsylvania
• 依托单位国家: 美国
• 项目类别: Continuing Grant
• 财政年份: 2021
• 资助国家: 美国
• 起止时间: 2021-10-01 至 2024-09-30
• 项目状态: 已结题
• 来源: https://www.nsf.gov/awardsearch/showAward?AWD_ID=2124355&HistoricalAwards=false
• 关键词: NCS; FO; Tracking; social; behavior

Advances in technology, mathematics, computing and engineering are making it possible to quantify behaviors within complex naturalistic environments and to relate them to underlying neural mechanisms. For social animals, which have evolved to perceive and evaluate signals within a community context, the ability to link neural function with the precise social environment is especially important and challenging. Little is currently known about how the brain integrates complex social information and how such information might be encoded. This stems in part from the experimental challenge of measuring and assessing the variables that determine a social context and then linking the state of a social network to precise neural events. This project has assembled an interdisciplinary team of engineers, neurobiologists and computational scientists to create a platform to record and evaluate brain dynamics in individual animals navigating a complex social environment. In addition to the challenge and opportunity of using sophisticated engineering and computational approaches to study how brains encode social information, this work will generate a complex dataset that will offer unique opportunities for developing novel mathematical methods to quantify and visualize social networks that can be applied to other disciplines.In this study, a "smart aviary" is equipped with arrays of cameras and microphones to create a fully automated system for tracking moment-to-moment behavioral events for each individual songbird within a social group. The songbirds are of a highly gregarious species (brown-headed cowbird, Molothrus ater) that uses vocal communications to form and maintain a complex social system. As a variable, social context needs to be mathematically constructed over multiple timescales from the sequence of all behavioral events. This entails the development of new mathematical approaches and statistical models for quantifying social network state so that individual neural events can be linked back to the precise social contexts. The project will develop new machine learning approaches for automated capture of social interactions over months-long time periods. In addition, an articulated mesh model enables visual signals to be captured in unprecedented detail, allowing the quantification of shape-mediated social signaling.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.

技术、数学、计算和工程学的进步使得量化复杂自然环境中的行为并将其与潜在的神经机制联系起来成为可能。对于社会性动物来说，它们已经进化到在社区环境中感知和评估信号，将神经功能与精确的社会环境联系起来的能力尤其重要和具有挑战性。关于大脑如何整合复杂的社会信息以及这些信息如何编码，目前还知之甚少。这部分源于测量和评估决定社会背景的变量，然后将社会网络的状态与精确的神经事件联系起来的实验挑战。该项目组建了一个由工程师、神经生物学家和计算科学家组成的跨学科团队，旨在创建一个平台，记录和评估在复杂社会环境中航行的个体动物的大脑动力学。除了使用复杂的工程和计算方法来研究大脑如何编码社会信息的挑战和机遇外，这项工作还将生成一个复杂的数据集，为开发新的数学方法提供独特的机会，以量化和可视化可应用于其他学科的社交网络。在这项研究中，“智能鸟舍”配备有摄像机和麦克风阵列，以创建用于跟踪社会群体内每只个体鸣禽的每时每刻行为事件的全自动系统。鸣禽是一种高度群居的物种（棕头燕八哥，Molothrus ater），使用声音通信来形成和维持复杂的社会系统。作为一个变量，社会背景需要从所有行为事件的序列中在多个时间尺度上进行数学构建。这需要开发新的数学方法和统计模型来量化社交网络状态，以便将单个神经事件与精确的社交环境联系起来。该项目将开发新的机器学习方法，用于在长达数月的时间内自动捕获社交互动。此外，铰接式网格模型能够捕捉到前所未有的细节视觉信号，从而量化形状介导的社会信号。该奖项反映了NSF的法定使命，并通过使用基金会的智力价值和更广泛的影响审查标准进行评估，被认为值得支持。

项目成果

Birds of a Feather: Capturing Avian Shape Models from Images

• DOI: 10.1109/cvpr46437.2021.01450
• 发表时间: 2021-05
• 期刊: 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)
• 作者: Yufu Wang;Nikos Kolotouros;Kostas Daniilidis;M. Badger

Multi-view Tracking, Re-ID, and Social Network Analysis of a Flock of Visually Similar Birds in an Outdoor Aviary

• DOI: 10.1007/s11263-023-01768-z
• 发表时间: 2023-03-06
• 期刊: INTERNATIONAL JOURNAL OF COMPUTER VISION
• 影响因子: 19.5
• 作者: Xiao，Shiting;Wang，Yufu;Badger，Marc
• 通讯作者: Badger，Marc