Collaborative Research: LightningBug, An Integrated Pipeline to Overcome The Biodiversity Digitization Gap

昆虫是我们星球上最大和最多样化的动物，它们在生态系统和为社会提供的服务中发挥着重要作用。长期以来，昆虫学家一直致力于收集、保存和存放在地球仪的自然历史博物馆中的近10亿个昆虫标本。这些标本构成了我们对昆虫知识的基础，并提供了有关过去的关键信息，科学家可以从中评估当前和未来的全球变化影响。为了充分实现这些收藏品的价值，昆虫标本的数据必须首先数字化。然而，它们体积小，结构精致，传统的存储和标签方法为大规模数字化带来了巨大的挑战。因此，目前，只有5%的标本有转录标签和不到1%的标本成像。LightningBug项目将通过建立一个半自动化的工作流程来突破这一数字化瓶颈，该工作流程涉及机器人多视图成像、信息提取和3D重建方面的进步。这项工作的结果将为研究人员提供前所未有的能力，以捕捉标本元数据代表时间，地点和分类身份沿着与准确的三维表面形态代表颜色和形状。我们希望LightningBug和相关技术将促进生态形态学研究的规模，迄今为止还不可能。LightningBug项目旨在创建一个端到端的管道，用于从昆虫标本中的固定昆虫进行高通量数据采集。为了实现这一目标，我们将：（1）进一步开发现有的硬件和软件平台，以捕获标签和标本的多视图图像;（2）建立强大的算法，以自动将多个标签的片段视图处理为单独的集成“虚拟标签”;（3）将虚拟标签连接到结构化文本提取服务;（4）应用摄影测量分析技术对试件的三维形状和结构进行组合。在现实世界科学用例的指导下，强调在全球变化和功能形态学研究中使用基于多视图成像的多视图成像，耶鲁皮博迪博物馆和哈佛比较动物学博物馆的昆虫学藏品将用于严格的测试用例实现。结果将包括注释的多视图图像，3D模型的标本（点云，纹理网格），2D重建的“虚拟标签”和数字化的标本元数据从这些标签生成的强大的集。这些数字标本将为数据保存和访问带来新的挑战，但它们也将促进大规模存储和交付研究图像的新解决方案。这一挑战将通过与MorphoSource建立伙伴关系来解决，以开发一个链接的机构存储库模型，用于访问大型数字资产的数据，例如通过多视图成像产生的数据。最终，捕获多视图图像套件并大规模生成虚拟标本的能力将为远程访问研究资源提供新的途径，并将计算机视觉和机器学习应用于性状识别和进化，物种识别和新物种发现。来自被钉死昆虫的标签数据将使研究人员能够获得将生物多样性变化与其他生物和环境变量联系起来所需的关键时间和地理空间信息。它还将为收藏工作人员提供其藏品的完整数字画像，这可以进行历史研究，简化藏品的使用和跟踪，并改善数据质量控制。该项目的成果还将应用于自然历史收藏和研究社区以外的领域，如计算机图形学、产品成像、电影、3D动画、虚拟现实和增强现实以及教育。该项目的更多信息和结果可以在www.example.com上找到http://lightningbug.techThis奖项反映了NSF的法定使命，并被认为值得通过使用基金会的知识价值和更广泛的影响审查标准进行评估来支持。

Insects are the largest and most diverse class of animals on our planet where they play essential roles in ecosystems and the services those provide to society. Entomologists have long been engaged in collecting, preserving and depositing nearly one billion insect specimens at natural history museums around the globe. These collections form the basis for much of our knowledge about insects and provide critical information about the past from which scientists can assess current and future global change impacts. To fully realize the value of these collections, data from insect specimens must first be digitized. However, their small size, delicate structures, and traditional storage and labeling methods creates enormous challenges for large-scale digitization. Consequently, at present, only 5% of specimens have transcribed labels and less than 1% of specimens are imaged. The LightningBug project will break through this digitization bottleneck by establishing a semi-automated workflow involving advancements in robotic multi-view imaging, information extraction and 3D reconstruction. Results from this work will provide researchers with the unprecedented capability to capture specimen metadata representing time, place and taxonomic identity along with accurate three-dimensional surface morphology representing color and shape. We expect LightningBug and related technologies will promote ecomorphological studies at a scale that has not been possible to date.The LightningBug project seeks to create an end-to-end pipeline for high-throughput data acquisition from pinned insects in entomological collections. To accomplish this goal, we will: (1) further develop an existing hardware and software platform to capture multi-view imagery of both labels and specimens; (2) build robust algorithms to automatically process fragmentary views of multiple labels into separate integrated “virtual labels;" (3) connect virtual labels to structured text extraction services; and (4) apply photogrammetric analysis to assemble the 3D shape and structure of specimens. Guided by real-world science use cases that highlight the use of specimen-based multi-view imaging in studies of global change and functional morphology, the entomological collections of the Yale Peabody Museum and the Harvard Museum of Comparative Zoology will be used in rigorous test-case implementations. Results will include robust sets of annotated multi-view images, 3D models of specimens (point clouds, textured meshes), 2D reconstructed “virtual labels” and digitized specimen metadata generated from those labels. These digital specimens will present new challenges for data preservation and access, but they will also catalyze new solutions for large-scale storage and delivery of research imagery. This challenge will be addressed via a partnership with MorphoSource to develop a linked institutional repository model for data access to large digital assets such as those produced by multi-view imaging. Ultimately, the ability to capture multi-view image suites and generate virtual specimens at scale will permit new avenues for remote access to research resources, and enable the application of computer vision and machine learning to trait identification and evolution, species recognition and new species discovery. Label data from pinned insects will give researchers access to critical temporal and geospatial information necessary for relating changes in biodiversity to other biotic and environmental variables. It will also provide collections staff with a complete digital portrait of their holdings, which can enable historical research, streamline collections use and tracking, and improve data quality control. Results from this project will also have applications beyond the natural history collections and research communities, such as computer graphics, product imaging, motion pictures, 3D animation, virtual and augmented realities, and education. More information and results from this project can be found at http://lightningbug.techThis award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.