MRI: Development of an Observatory for Quantitative Analysis of Collective Behavior in Animals

该项目开发了一种新的仪器，可以精确定量分析动物在真实的世界场景中的运动和声音表达，旨在促进复杂现实环境中动物行为和神经科学研究的创新研究。虽然在研究行为的大脑机制方面取得了很大进展，但这些进展主要限于在相对简单的环境中研究个体受试者。对于包括人类在内的许多社会性物种来说，理解这些更复杂环境范围内的神经生物学过程至关重要，因为它们的大脑已经进化到在社会背景下感知和评估信号。事实上，今天在视频捕获硬件和存储以及计算机视觉和网络科学算法方面的进步使这种动物促进成为可能。过去的工作依赖于主观和耗时的观察，从视频流，遭受不精确，低维度，和专家分析师的感官辨别力的限制。该仪器不仅将自动化检测行为的过程，而且还为社会群体中的每个人提供时间和空间上的精确数字表征。虽然没有明确的仪器的一部分，由我们的系统提供的定量描述将允许相关的社会背景与神经测量的能力，一个任务，可能只有当足够的时空精度已经实现时才能完成。该仪器使行为和神经科学的研究和计算机视觉和网络理论的新算法的发展。在行为科学中，仪器允许在动物或人类的小群体中生成社会行为的网络模型，这些模型可以用来提出问题，这些问题可以从网络的动态如何影响性选择，生殖成功，甚至健康信息，到个人的声音决策如何引起社会统治等级。在神经科学中，系统提供的精确时空信息可用于评估在精确定义的社会背景下感觉处理和行为决策的神经基础。例如，对给定的声音刺激的感官反应可以通过动物听到刺激的背景以及他和发送者在群体中的先前行为历史来评估。在计算机视觉中，我们提出了新的方法来校准多个摄像机“在野外”，结合外观和几何形状从视频中提取精确的3D姿势和身体部位，学习一组动物之间的注意力集中，以及声源的估计和发声的分类。新的方法将被用于分层发现的行为图，将相互作用超出了成对的水平与单纯复杂，和一个新的理论图动力学的时间演化的社会行为。该仪器使行为和神经科学家受益。因此，开发的代码和算法将是开源的，以便科学界可以根据应用程序对其进行扩展。拟议的工作也影响计算机视觉和网络科学，因为设计的基本算法应该推进最先进的技术水平。对于其他计算机视觉算法的性能评估，将采用已建立的数据集。

This project, developing a new instrument to enable an accurate quantitative analysis of the movement of animals and vocal expressions in real world scenes, aims to facilitate innovative research in the study of animal behavior and neuroscience in complex realistic environments. While much progress has been made investigating brain mechanisms of behavior, these have been limited primarily to studying individual subjects in relatively simple settings. For many social species, including humans, understanding neurobiological processes within the confines of these more complex environments is critical because their brains have evolved to perceive and evaluate signals within a social context. Indeed, today's advances in video capture hardware and storage and in algorithms in computer vision and network science make this facilitation with animals possible. Past work has relied on subjective and time-consuming observations from video streams, which suffer from imprecision, low dimensionality, and the limitations of the expert analyst's sensory discriminability. This instrument will not only automate the process of detecting behaviors but also provide an exact numeric characterization in time and space for each individual in the social group. While not explicitly part of the instrument, the quantitative description provided by our system will allow the ability to correlate social context with neural measurements, a task that may only be accomplished when sufficient spatiotemporal precision has been achieved.The instrument enables research in the behavioral and neural sciences and development of novel algorithms in computer vision and network theory. In the behavioral sciences, the instrumentation allows the generation of network models of social behavior in small groups of animals or humans that can be used to ask questions that can range from how the dynamics of the networks influence sexual selection, reproductive success, and even health messaging to how vocal decision making in individuals gives rise to social dominance hierarchies. In the neural sciences, the precise spatio-temporal information the system would provide can be used to evaluate the neural bases of sensory processing and behavioral decision under precisely defined social contexts. Sensory responses to a given vocal stimulus, for example, can be evaluated by the context in which the animal heard the stimulus and both his and the sender's prior behavioral history in the group. In computer vision, we propose novel approaches for the calibration of multiple cameras "in the wild", the combination of appearance and geometry for the extraction of exact 3D pose and body parts from video, the learning of attentional focus among animals in a group, and the estimation of sound source and the classification of vocalizations. New approaches will be used on hierarchical discovery of behaviors in graphs, the incorporation of interactions beyond the pairwise level with simplicial complices, and a novel theory of graph dynamics for the temporal evolution of social behavior. The instrumentation benefits behavioral and neural scientists. Therefore, the code and algorithms developed will be open-source so that the scientific community can extend them based on the application. The proposed work also impacts computer vision and network science because the fundamental algorithms designed should advance the state of the art. For performance evaluation of other computer vision algorithms, established datasets will be employed.