IIBR Informatics: Automatic generation of multi-layered, information-rich 3D maps of ecosystems from images

生态系统图提供了关于物种及其环境之间的丰度、分布和空间关系的信息，提供了对产生观察模式的基本生态过程（竞争、捕食、促进等）的见解。长期以来，卫星和飞机的远程成像一直被用于绘制生态系统和栖息地的分布图，但这些来源的图像通常没有足够的分辨率来识别单个生物体，限制了它们在研究种群和群落方面的应用。该项目将以计算机视觉和人工智能的最新发展为基础，允许用消费级相机拍摄的“本地”图像组装成生态系统的三维地图。将开发主要基于深度神经网络的工具，以自动绘制物种和基质、个体和生物群落的分布。这些新开发的方法将通过提供快速、自动化的方法来绘制生态系统，从而使人们能够对生物之间的空间模式和关系有新的认识，而不是目前手工、实验室密集型的绘制方法。该项目将支持几名研究生和本科生，并将参与所有年龄的公民科学家通过网络游戏。生态系统的时空格局是环境变化、资源限制、竞争和捕食等基本生态过程的产物。在这个项目中，将开发利用当地图像对生态系统进行高分辨率（毫米到厘米）测绘的工具，从而快速评估生物的丰度和分布，以及生物之间的空间关系。这些工具将整合深度学习架构的最新进展，如卷积神经网络（cnn）和用于组装“局部”图像的算法，以自动生成生态系统的三维（3D）信息丰富的地图。商业和开源软件目前可用于将重叠的2D图像自动组装成图像场景的3D重建。本文开发的方法将在基于CNN的图像分析方法的三维重建上叠加相应图像中编码的相关生物信息。该项目的具体目标是：1)对重建的3D地图进行语义分割，即将3D地图上的每个点标记为多个预定义类别之一；2)通过物体检测器识别个体；3)使用无监督主题模型绘制社区；4)使用运动的非刚性结构将移动的生物体结合起来，从而生成信息丰富的多层生态系统3D地图。本项目开发的工具一般适用于大多数生态系统；然而，珊瑚礁和盐沼将被用作范例生态系统，在生物多样性、3D结构复杂性和成像条件方面具有相关的对比。该项目的结果将在Hopkinson实验室网站（www.hopkinsonlab.org）上公布，代码将在Github上公开（https://github.com/bmhopkinson）.This)，该奖项反映了NSF的法定使命，并通过使用基金会的智力价值和更广泛的影响审查标准进行评估，被认为值得支持。

Ecosystem maps provide information on the abundance, distribution, and spatial relationships between species and their environment, providing insight into the fundamental ecological processes (competition, predation, facilitation, etc) that generate observed patterns. Remote imaging from satellites and airplanes have long been used to map the distribution of ecosystems and habitats, but typically the imagery from these sources does not have sufficient resolution to identify individual organisms limiting their use for studying populations and communities. This project will build on recent developments in computer vision and artificial intelligence that allow 'local' images taken with consumer-grade cameras to be assembled into three dimensional maps of ecosystems. Tools, based primarily on deep-neural networks, will be developed to automatically map the distribution of species and substrates, individuals, and biological communities. These newly developed approaches will enable novel insights into spatial patterns and relationships among organisms by providing rapid, automated methods to map ecosystems, in contrast to current manual, lab-intensive mapping methods. The project will support several graduate and undergraduate students and will involve citizen scientists of all ages through web-based games. Spatio-temporal patterns in ecosystems are the product of fundamental ecological processes such asenvironmental variation, resource limitation, competition, and predation. In this project, tools will be developed for high-resolution (mm to cm) mapping of ecosystems using local imagery allowing rapid assessment of the abundance and distribution of organisms, and spatial relationships amongst organisms. These tools will integrate recent advances in deep learning architectures such as convolutional neural networks (CNNs) and algorithms for assembly of 'local' images to automatically generate three-dimensional (3D) information-rich maps of ecosystems. Commercial and open-source software are currently available for automated assembly of overlapping 2D images into 3D reconstructions of the imaged scene. The methods develop here will overlay relevant biological information encoded in the corresponding images upon 3D reconstructions using CNN based image analysis methods. The specific goals of the project are: 1) semantic segmentation of the reconstructed 3D maps, i.e., labeling of each point on the 3D map as one of a number of predefined classes, 2) identification of individuals via object detectors, 3) mapping of communities using unsupervised topic models, and 4) incorporation of moving organisms using non-rigid structure from motion, resulting in an information-rich, multi-layered 3D map of the ecosystem. The tools developed in this project will be generally applicable to most ecosystems; however, coral reefs and salt marshes will be used as exemplar ecosystems with relevant contrasts in biodiversity, 3D structural complexity, and imaging conditions. The results of the project will be made available on the Hopkinson lab website (www.hopkinsonlab.org) and code will be made publicly available on Github (https://github.com/bmhopkinson).This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.