MRI: Development of an Observatory for Quantitative Analysis of Collective Behavior in Animals

该项目开发了一种新的仪器，能够对真实世界场景中动物的运动和声音表达进行准确的量化分析，旨在促进在复杂现实环境中研究动物行为和神经科学的创新研究。虽然对行为的大脑机制的研究已经取得了很大进展，但这些主要局限于在相对简单的环境中研究单个受试者。对于包括人类在内的许多社会物种来说，在这些更复杂的环境中了解神经生物学过程是至关重要的，因为他们的大脑已经进化到在社会环境中感知和评估信号。事实上，今天在视频捕获硬件和存储方面的进步，以及计算机视觉和网络科学中的算法的进步，使动物的这种便利成为可能。过去的工作依赖于对视频流的主观和耗时的观察，这些观察受到不精确、低维和专家分析师感官辨别能力的限制。这一工具不仅将自动检测行为的过程，而且还将为社会群体中的每个人提供准确的时间和空间数字表征。虽然不是该仪器的明确组成部分，但我们的系统提供的定量描述将允许将社会背景与神经测量相关联的能力，这一任务可能只有在实现足够的时空精度时才能完成。该仪器使行为和神经科学的研究以及计算机视觉和网络理论中的新算法的开发成为可能。在行为科学中，这种工具允许在一小群动物或人类中产生社会行为的网络模型，这些模型可以用来提出各种问题，从网络的动态如何影响性选择、生殖成功，甚至是健康信息，到个人的直言不讳的决策如何产生社会支配等级。在神经科学中，该系统将提供的精确时空信息可以用来评估在精确定义的社会背景下感觉处理和行为决策的神经基础。例如，对给定发声刺激的感觉反应可以通过动物听到刺激的背景以及动物和发送者在群体中的先前行为史来评估。在计算机视觉中，我们提出了新的方法来校准多个摄像机，结合外观和几何从视频中提取准确的3D姿势和身体部位，学习一组动物的注意焦点，以及声源的估计和发声的分类。新的方法将用于图中行为的分层发现，用单纯复数将两两级别以外的交互结合在一起，以及用于社会行为的时间演化的新的图动力学理论。这种仪器使行为和神经科学家受益。因此，开发的代码和算法将是开源的，这样科学界就可以在应用程序的基础上扩展它们。这项拟议的工作还影响了计算机视觉和网络科学，因为所设计的基本算法应该会推动最先进的技术。对于其他计算机视觉算法的性能评估，将使用已建立的数据集。

This project, developing a new instrument to enable an accurate quantitative analysis of the movement of animals and vocal expressions in real world scenes, aims to facilitate innovative research in the study of animal behavior and neuroscience in complex realistic environments. While much progress has been made investigating brain mechanisms of behavior, these have been limited primarily to studying individual subjects in relatively simple settings. For many social species, including humans, understanding neurobiological processes within the confines of these more complex environments is critical because their brains have evolved to perceive and evaluate signals within a social context. Indeed, today's advances in video capture hardware and storage and in algorithms in computer vision and network science make this facilitation with animals possible. Past work has relied on subjective and time-consuming observations from video streams, which suffer from imprecision, low dimensionality, and the limitations of the expert analyst's sensory discriminability. This instrument will not only automate the process of detecting behaviors but also provide an exact numeric characterization in time and space for each individual in the social group. While not explicitly part of the instrument, the quantitative description provided by our system will allow the ability to correlate social context with neural measurements, a task that may only be accomplished when sufficient spatiotemporal precision has been achieved.The instrument enables research in the behavioral and neural sciences and development of novel algorithms in computer vision and network theory. In the behavioral sciences, the instrumentation allows the generation of network models of social behavior in small groups of animals or humans that can be used to ask questions that can range from how the dynamics of the networks influence sexual selection, reproductive success, and even health messaging to how vocal decision making in individuals gives rise to social dominance hierarchies. In the neural sciences, the precise spatio-temporal information the system would provide can be used to evaluate the neural bases of sensory processing and behavioral decision under precisely defined social contexts. Sensory responses to a given vocal stimulus, for example, can be evaluated by the context in which the animal heard the stimulus and both his and the sender's prior behavioral history in the group. In computer vision, we propose novel approaches for the calibration of multiple cameras "in the wild", the combination of appearance and geometry for the extraction of exact 3D pose and body parts from video, the learning of attentional focus among animals in a group, and the estimation of sound source and the classification of vocalizations. New approaches will be used on hierarchical discovery of behaviors in graphs, the incorporation of interactions beyond the pairwise level with simplicial complices, and a novel theory of graph dynamics for the temporal evolution of social behavior. The instrumentation benefits behavioral and neural scientists. Therefore, the code and algorithms developed will be open-source so that the scientific community can extend them based on the application. The proposed work also impacts computer vision and network science because the fundamental algorithms designed should advance the state of the art. For performance evaluation of other computer vision algorithms, established datasets will be employed.