Collaborative Research: LightningBug, An Integrated Pipeline to Overcome The Biodiversity Digitization Gap

昆虫是地球上最大、种类最多的动物，它们在生态系统和为社会提供的服务中发挥着至关重要的作用。昆虫学家长期以来一直致力于收集、保存和存放在全球自然历史博物馆的近10亿个昆虫标本。这些收集形成了我们对昆虫的大部分知识的基础，并提供了关于过去的关键信息，科学家可以从中评估当前和未来的全球变化影响。为了充分实现这些收藏的价值，昆虫标本的数据必须首先数字化。然而，它们的体积小，结构精致，传统的存储和标签方法为大规模数字化带来了巨大的挑战。因此，目前只有5%的标本进行了转录标记，不到1%的标本进行了成像。LightningBug项目将通过建立涉及机器人多视图成像、信息提取和3D重建的半自动化工作流程来突破这一数字化瓶颈。这项工作的结果将为研究人员提供前所未有的能力，以捕获代表时间、地点和分类身份的标本元数据，以及代表颜色和形状的精确三维表面形态。我们期待lightening bug和相关技术将在迄今为止不可能的规模上促进生态形态学研究。闪电虫项目旨在建立一个端到端的管道，从昆虫学收集的固定昆虫中获取高通量数据。为了实现这一目标，我们将：(1)进一步开发现有的硬件和软件平台，以捕获标签和标本的多视图图像；(2)构建鲁棒算法，将多个标签的碎片视图自动处理成独立的集成“虚拟标签”；(3)将虚拟标签与结构化文本提取服务连接起来；(4)应用摄影测量分析对标本的三维形状和结构进行组装。在现实科学用例的指导下，强调在全球变化和功能形态学研究中使用基于标本的多视图成像，耶鲁皮博迪博物馆和哈佛比较动物学博物馆的昆虫学收藏品将用于严格的测试用例实施。结果将包括鲁棒的注释多视图图像集，标本的3D模型（点云，纹理网格），二维重建的“虚拟标签”以及由这些标签生成的数字化标本元数据。这些数字标本将为数据保存和访问带来新的挑战，但它们也将催生大规模存储和交付研究图像的新解决方案。这一挑战将通过与MorphoSource的合作来解决，以开发一个链接的机构存储库模型，用于访问大型数字资产（如由多视图成像产生的数据）。最终，捕获多视图图像套件和大规模生成虚拟标本的能力将为远程访问研究资源提供新的途径，并使计算机视觉和机器学习应用于特征识别和进化，物种识别和新物种发现。来自固定昆虫的标签数据将使研究人员能够获得将生物多样性变化与其他生物和环境变量联系起来所必需的关键时间和地理空间信息。它还将为馆藏工作人员提供其馆藏的完整数字画像，从而可以进行历史研究，简化馆藏的使用和跟踪，并改善数据质量控制。该项目的成果也将应用于自然历史收藏和研究领域以外的领域，如计算机图形学、产品成像、电影、3D动画、虚拟和增强现实以及教育。该项目的更多信息和结果可在http://lightningbug.techThis上找到，该奖项反映了美国国家科学基金会的法定使命，并通过使用基金会的智力价值和更广泛的影响审查标准进行评估，被认为值得支持。

Insects are the largest and most diverse class of animals on our planet where they play essential roles in ecosystems and the services those provide to society. Entomologists have long been engaged in collecting, preserving and depositing nearly one billion insect specimens at natural history museums around the globe. These collections form the basis for much of our knowledge about insects and provide critical information about the past from which scientists can assess current and future global change impacts. To fully realize the value of these collections, data from insect specimens must first be digitized. However, their small size, delicate structures, and traditional storage and labeling methods creates enormous challenges for large-scale digitization. Consequently, at present, only 5% of specimens have transcribed labels and less than 1% of specimens are imaged. The LightningBug project will break through this digitization bottleneck by establishing a semi-automated workflow involving advancements in robotic multi-view imaging, information extraction and 3D reconstruction. Results from this work will provide researchers with the unprecedented capability to capture specimen metadata representing time, place and taxonomic identity along with accurate three-dimensional surface morphology representing color and shape. We expect LightningBug and related technologies will promote ecomorphological studies at a scale that has not been possible to date.The LightningBug project seeks to create an end-to-end pipeline for high-throughput data acquisition from pinned insects in entomological collections. To accomplish this goal, we will: (1) further develop an existing hardware and software platform to capture multi-view imagery of both labels and specimens; (2) build robust algorithms to automatically process fragmentary views of multiple labels into separate integrated “virtual labels;" (3) connect virtual labels to structured text extraction services; and (4) apply photogrammetric analysis to assemble the 3D shape and structure of specimens. Guided by real-world science use cases that highlight the use of specimen-based multi-view imaging in studies of global change and functional morphology, the entomological collections of the Yale Peabody Museum and the Harvard Museum of Comparative Zoology will be used in rigorous test-case implementations. Results will include robust sets of annotated multi-view images, 3D models of specimens (point clouds, textured meshes), 2D reconstructed “virtual labels” and digitized specimen metadata generated from those labels. These digital specimens will present new challenges for data preservation and access, but they will also catalyze new solutions for large-scale storage and delivery of research imagery. This challenge will be addressed via a partnership with MorphoSource to develop a linked institutional repository model for data access to large digital assets such as those produced by multi-view imaging. Ultimately, the ability to capture multi-view image suites and generate virtual specimens at scale will permit new avenues for remote access to research resources, and enable the application of computer vision and machine learning to trait identification and evolution, species recognition and new species discovery. Label data from pinned insects will give researchers access to critical temporal and geospatial information necessary for relating changes in biodiversity to other biotic and environmental variables. It will also provide collections staff with a complete digital portrait of their holdings, which can enable historical research, streamline collections use and tracking, and improve data quality control. Results from this project will also have applications beyond the natural history collections and research communities, such as computer graphics, product imaging, motion pictures, 3D animation, virtual and augmented realities, and education. More information and results from this project can be found at http://lightningbug.techThis award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.