MRI: Development of an Observatory for Quantitative Analysis of Collective Behavior in Animals

该项目旨在开发一种新的仪器，能够对真实世界场景中的动物运动和声音表达进行准确的定量分析，旨在促进复杂现实环境中动物行为和神经科学研究的创新研究。虽然对大脑行为机制的研究取得了很大进展，但这些研究主要局限于在相对简单的环境下研究个体。对于包括人类在内的许多社会物种来说，在这些更复杂的环境范围内理解神经生物学过程至关重要，因为他们的大脑已经进化到能够感知和评估社会环境中的信号。事实上，今天在视频捕捉硬件和存储以及计算机视觉和网络科学算法方面的进步使这种与动物的便利成为可能。过去的工作依赖于对视频流的主观和耗时的观察，这些观察受到不精确、低维度和专家分析师感官辨别能力的限制的影响。这一工具不仅将自动检测行为的过程，而且还将在时间和空间上为社会群体中的每个个体提供精确的数字特征。虽然不是仪器的明确组成部分，但我们的系统提供的定量描述将允许将社会背景与神经测量相关联的能力，这一任务只有在达到足够的时空精度时才能完成。该仪器使行为科学和神经科学的研究以及计算机视觉和网络理论的新算法的发展成为可能。在行为科学中，仪器允许在动物或人类的小群体中产生社会行为的网络模型，这些模型可以用来提出各种问题，从网络的动态如何影响性选择，繁殖成功，甚至健康信息，到个人的声音决策如何产生社会统治等级。在神经科学中，该系统提供的精确时空信息可用于评估在精确定义的社会环境下感觉加工和行为决策的神经基础。例如，对特定声音刺激的感觉反应可以通过动物听到刺激的环境以及他和发出者之前在群体中的行为历史来评估。在计算机视觉方面，我们提出了新的方法来校准“野外”的多个摄像机，结合外观和几何形状从视频中提取精确的3D姿势和身体部位，学习一群动物之间的注意力焦点，以及估计声源和分类发声。新方法将用于图中行为的分层发现，结合超越两两水平的简单复合体的相互作用，以及社会行为时间进化的新图动力学理论。该仪器使行为和神经科学家受益。因此，开发的代码和算法将是开源的，以便科学界可以根据应用程序对其进行扩展。提议的工作也会影响计算机视觉和网络科学，因为所设计的基本算法应该推动最先进的技术。对于其他计算机视觉算法的性能评估，将使用已建立的数据集。

This project, developing a new instrument to enable an accurate quantitative analysis of the movement of animals and vocal expressions in real world scenes, aims to facilitate innovative research in the study of animal behavior and neuroscience in complex realistic environments. While much progress has been made investigating brain mechanisms of behavior, these have been limited primarily to studying individual subjects in relatively simple settings. For many social species, including humans, understanding neurobiological processes within the confines of these more complex environments is critical because their brains have evolved to perceive and evaluate signals within a social context. Indeed, today's advances in video capture hardware and storage and in algorithms in computer vision and network science make this facilitation with animals possible. Past work has relied on subjective and time-consuming observations from video streams, which suffer from imprecision, low dimensionality, and the limitations of the expert analyst's sensory discriminability. This instrument will not only automate the process of detecting behaviors but also provide an exact numeric characterization in time and space for each individual in the social group. While not explicitly part of the instrument, the quantitative description provided by our system will allow the ability to correlate social context with neural measurements, a task that may only be accomplished when sufficient spatiotemporal precision has been achieved.The instrument enables research in the behavioral and neural sciences and development of novel algorithms in computer vision and network theory. In the behavioral sciences, the instrumentation allows the generation of network models of social behavior in small groups of animals or humans that can be used to ask questions that can range from how the dynamics of the networks influence sexual selection, reproductive success, and even health messaging to how vocal decision making in individuals gives rise to social dominance hierarchies. In the neural sciences, the precise spatio-temporal information the system would provide can be used to evaluate the neural bases of sensory processing and behavioral decision under precisely defined social contexts. Sensory responses to a given vocal stimulus, for example, can be evaluated by the context in which the animal heard the stimulus and both his and the sender's prior behavioral history in the group. In computer vision, we propose novel approaches for the calibration of multiple cameras "in the wild", the combination of appearance and geometry for the extraction of exact 3D pose and body parts from video, the learning of attentional focus among animals in a group, and the estimation of sound source and the classification of vocalizations. New approaches will be used on hierarchical discovery of behaviors in graphs, the incorporation of interactions beyond the pairwise level with simplicial complices, and a novel theory of graph dynamics for the temporal evolution of social behavior. The instrumentation benefits behavioral and neural scientists. Therefore, the code and algorithms developed will be open-source so that the scientific community can extend them based on the application. The proposed work also impacts computer vision and network science because the fundamental algorithms designed should advance the state of the art. For performance evaluation of other computer vision algorithms, established datasets will be employed.