CyberSEES: Type 1: Collaborative Research: High-Performance Image Classification and Search Supporting Large-Scale Seafloor Biodiversity and Habitat Surveys

海底生态系统是由多种生物组成的复杂环境。不幸的是，这些生态系统日益受到直接和间接的人类活动的威胁，包括土地利用做法的变化、沿海径流、能源和矿物开采以及捕捞压力。制定有效的可持续性政策来应对这些生态系统威胁，需要我们首先了解海底群落的现状，然后跟踪它们随着人类活动的转变和可持续性政策的修改而如何随时间变化。高分辨率水下成像的最新进展为实现这一目标提供了新的途径。勘测船可以拖着水下摄像系统在受威胁的海域来回穿梭，反复拍摄海底的照片。这产生了一组巨大而有价值的图像，捕捉了海底生态系统的现状。像这样的调查已经在许多受威胁的地区进行了，更多的调查正在进行中。处理这些图像集仍然面临着重大挑战。要对海底栖息地进行有用的描述，就需要知道存在哪些特定类型的珊瑚、海绵、海星等，有多少，以及它们在整个地区的分布情况。但是，由于每个调查图像集包含数十万或数百万张图像，人工处理是不切实际的。计算机软件可以扫描每张图像，并自动识别不同海底物种的颜色和纹理，而不是由一群专家来检查这些图像。像这样的实验分类软件目前存在于研究实验室中，但软件速度很慢。为了对巨大的图像集有用，这个软件必须经过修改和优化，才能在最新的高性能超级计算机上运行。这是该项目的重点，它将产生新的优化分类软件，可以快速扫描大量的图像集来分类和计数存在的物种，并提供有关受威胁的海底生态系统的健康和生物多样性的基本信息，或任何其他生态系统与合适的图像集。然后，当每隔几年对同一地区重复进行调查时，这种处理可以揭示重要的趋势，这些趋势记录了一个地区的健康状况以及旨在减轻对这些社区的持续威胁的新的可持续性政策的影响。本项目利用先前的工作原型海底图像分类算法。这些算法将调查图像分成小块，然后使用高维特征向量对每个块进行表征，该特征向量包括瓷砖中存在的颜色和纹理的度量，以及测量设备在捕获图像时收集的水温、盐度和深度数据。特征向量中的颜色是基于贴图的量化色相直方图来选择的，而纹理是通过亮度离散余弦变换（DCT）系数来表征的。然后将瓷砖的特征向量与大型分类库中已知物种的存储特征向量进行比较。使用库中的一组最近邻匹配进行基于概率的选择，产生对图像tile中所描述的物种的最佳猜测。在整个图像调查过程中，这个过程一幅接一幅地重复进行。分类性能在很大程度上取决于分类库的大小和用于图像块和库条目的特征向量的维数。本项目提高分类性能的方法使用了分类库的自定义k-d树搜索数据结构，以及指导和调优分类过程的领域知识。该项目首先采用新方法，在分类之前，通过使用广泛的调查特征，如覆盖的地理区域、水温和盐度、声学数据的海底类型等，筛选树。其他技术通过使用调查和库度量来优化特征向量的构建和匹配，以剔除和权衡向量组件（例如上下文色域和纹理细节减少，主成分分析来组合和权衡特征），通过使用库多样性的k-d树度量来减少最近邻集的大小，重构k-d树以提高常见情况搜索和缓存性能。并在大型计算集群中的多个线程、内核、处理器和节点之间并行化高效分类搜索。这些新方法有望大大提高分类性能，并对最新的大型调查图像集进行有效处理。

Seafloor ecosystems are complex environments populated by a great diversity of organisms. Unfortunately, these ecosystems are increasingly threatened by direct and indirect human activities, including changes in land-use practices, coastal runoff, energy and mineral extraction, and fishing pressure. Developing effective sustainability policies to deal with these ecosystem threats requires that we first understand seafloor communities as they are today, and then track how they change over time as human activities shift and sustainability policies are modified. Recent advances in high-resolution underwater imaging offer new ways to do this. Survey ships can zigzag back and forth above a threatened region, towing a submerged camera system that repeatedly snaps pictures of the seafloor. This produces an enormous and valuable image set that captures the current state of a seafloor ecosystem. Surveys like this have been done for many threatened regions, and more are in progress. Substantial challenges remain to process these image sets. A useful characterization of a seafloor habitat requires knowing which specific types of corals, sponges, starfish, and so forth are present, how many there are, and how they are distributed throughout a region. But with each survey image set containing hundreds of thousands or millions of images, manual processing is impractical. Instead of an army of experts examining these images, computer software can scan each image and automatically recognize the color and texture of different seafloor species. Experimental classification software like this exists today in research laboratories, but the software is slow. To be useful for huge image sets, this software must be revised and optimized to run on the latest high-performance supercomputers. This is the focus of the project, which will yield new optimized classification software that can quickly sweep through enormous image sets to classify and count the species present and provide essential information about the health and biodiversity of threatened seafloor ecosystems, or any other ecosystem with a suitable image set. Then, when surveys are repeated for the same region every few years, this processing can reveal important trends that document the health of a region and the impact of new sustainability policies that aim to mitigate continuing threats to these communities.This project leverages prior work prototyping seafloor image classification algorithms. These algorithms divide survey images into small tiles, then characterize each tile with a high-dimensionality feature vector that includes metrics on the colors and textures present in the tile, along with water temperature, salinity, and depth data collected by the survey apparatus at the moment the image was captured. Colors in the feature vector are chosen based upon a quantized hue histogram of the tile, while textures are characterized by luminance Discrete-Cosine-Transform (DCT) coefficients. A tile's feature vector is then compared against stored feature vectors for known species within a large classification library. A probability-based selection using a set of nearest-neighbor matches from the library yields a best guess for the species depicted in the image tile. This process is repeated tile after tile, image after image throughout an image survey. Classification performance is strongly a function of the classification library size and the dimensionality of feature vectors used for image tiles and library entries. This project's approach to improve classification performance uses a customized k-d-tree search data structure for the classification library, along with domain knowledge to guide and tune the classification process. The project begins with new methods to cull the tree, prior to classification, by using broad survey characteristics, such as the geographic region covered, water temperature and salinity, the sea bottom type from acoustic data, and so forth. Additional techniques optimize the construction and matching of feature vectors by using survey and library metrics to cull and weigh vector components (such as contextual color gamut and texture detail reduction, principal component analysis to combine and weigh features), reduce the nearest-neighbor set size by using k-d tree metrics on library diversity, restructure the k-d tree to improve common case search and cache performance, and parallelize the search for efficient classification across multiple threads, cores, processors, and nodes in a large compute cluster. Together these new methods are expected to substantially increase classification performance and enable efficient processing for the latest large survey image sets.