Collaborative Research: LightningBug, An Integrated Pipeline to Overcome The Biodiversity Digitization Gap

昆虫是地球上最大和最多样化的动物类别，它们在生态系统和为社会提供的服务中发挥着重要作用。昆虫学家长期以来一直致力于收集、保存和存放全球各地自然历史博物馆的近10亿份昆虫标本。这些收集形成了我们关于昆虫的大部分知识的基础，并提供了关于过去的关键信息，科学家可以从这些信息中评估当前和未来的全球变化影响。为了充分实现这些收藏的价值，首先必须将昆虫标本的数据数字化。然而，它们的体积小、结构精细，以及传统的存储和标签方法给大规模数字化带来了巨大的挑战。因此，目前只有5%的标本有转录标记，只有不到1%的标本被成像。LightningBug项目将通过建立一个半自动的工作流程来突破这一数字化瓶颈，该工作流程涉及机器人多视角成像、信息提取和3D重建方面的进步。这项工作的结果将为研究人员提供前所未有的能力，以获取代表时间、地点和分类身份的标本元数据，以及代表颜色和形状的准确三维表面形态。我们预计LightningBug和相关技术将以迄今无法实现的规模推动生态地貌研究。LightningBug项目寻求创建一条端到端的管道，用于从昆虫学收集中的固定昆虫中高通量获取数据。为了实现这一目标，我们将：(1)进一步开发现有的硬件和软件平台，以捕获标签和标本的多视图图像；(2)建立健壮的算法，将多个标签的零散视图自动处理为单独的集成“虚拟标签”；(3)将虚拟标签连接到结构化文本提取服务；以及(4)应用摄影测量分析来组装标本的3D形状和结构。在强调在全球变化和功能形态研究中使用基于标本的多视角成像的真实世界科学用例的指导下，耶鲁大学皮博迪博物馆和哈佛比较动物学博物馆的昆虫学收藏品将用于严格的测试用例实施。结果将包括一组健壮的带注释的多视角图像、标本的3D模型(点云、纹理网格)、2D重建的“虚拟标签”和由这些标签生成的数字化标本元数据。这些数字标本将对数据保存和获取提出新的挑战，但它们也将促进大规模存储和交付研究图像的新解决方案。这一挑战将通过与MorPhoSource建立伙伴关系来解决，该伙伴关系将开发一个链接的机构储存库模型，用于数据访问大型数字资产，例如由多视角成像产生的资产。最终，捕获多视角图像套件并大规模生成虚拟标本的能力将允许远程访问研究资源的新途径，并使计算机视觉和机器学习能够应用于特征识别和进化、物种识别和新物种发现。来自固定昆虫的标签数据将使研究人员获得关键的时间和地理空间信息，这些信息是将生物多样性的变化与其他生物和环境变量联系起来所必需的。它还将为藏品工作人员提供其藏品的完整数字肖像，从而能够进行历史研究，简化藏品的使用和跟踪，并改进数据质量控制。该项目的成果还将应用于自然历史收藏和研究社区之外，如计算机图形学、产品成像、电影、3D动画、虚拟和增强现实以及教育。该项目的更多信息和结果可以在http://lightningbug.techThis上找到，该奖项反映了国家科学基金会的法定使命，并通过使用基金会的智力优势和更广泛的影响审查标准进行评估，被认为值得支持。

Insects are the largest and most diverse class of animals on our planet where they play essential roles in ecosystems and the services those provide to society. Entomologists have long been engaged in collecting, preserving and depositing nearly one billion insect specimens at natural history museums around the globe. These collections form the basis for much of our knowledge about insects and provide critical information about the past from which scientists can assess current and future global change impacts. To fully realize the value of these collections, data from insect specimens must first be digitized. However, their small size, delicate structures, and traditional storage and labeling methods creates enormous challenges for large-scale digitization. Consequently, at present, only 5% of specimens have transcribed labels and less than 1% of specimens are imaged. The LightningBug project will break through this digitization bottleneck by establishing a semi-automated workflow involving advancements in robotic multi-view imaging, information extraction and 3D reconstruction. Results from this work will provide researchers with the unprecedented capability to capture specimen metadata representing time, place and taxonomic identity along with accurate three-dimensional surface morphology representing color and shape. We expect LightningBug and related technologies will promote ecomorphological studies at a scale that has not been possible to date.The LightningBug project seeks to create an end-to-end pipeline for high-throughput data acquisition from pinned insects in entomological collections. To accomplish this goal, we will: (1) further develop an existing hardware and software platform to capture multi-view imagery of both labels and specimens; (2) build robust algorithms to automatically process fragmentary views of multiple labels into separate integrated “virtual labels;" (3) connect virtual labels to structured text extraction services; and (4) apply photogrammetric analysis to assemble the 3D shape and structure of specimens. Guided by real-world science use cases that highlight the use of specimen-based multi-view imaging in studies of global change and functional morphology, the entomological collections of the Yale Peabody Museum and the Harvard Museum of Comparative Zoology will be used in rigorous test-case implementations. Results will include robust sets of annotated multi-view images, 3D models of specimens (point clouds, textured meshes), 2D reconstructed “virtual labels” and digitized specimen metadata generated from those labels. These digital specimens will present new challenges for data preservation and access, but they will also catalyze new solutions for large-scale storage and delivery of research imagery. This challenge will be addressed via a partnership with MorphoSource to develop a linked institutional repository model for data access to large digital assets such as those produced by multi-view imaging. Ultimately, the ability to capture multi-view image suites and generate virtual specimens at scale will permit new avenues for remote access to research resources, and enable the application of computer vision and machine learning to trait identification and evolution, species recognition and new species discovery. Label data from pinned insects will give researchers access to critical temporal and geospatial information necessary for relating changes in biodiversity to other biotic and environmental variables. It will also provide collections staff with a complete digital portrait of their holdings, which can enable historical research, streamline collections use and tracking, and improve data quality control. Results from this project will also have applications beyond the natural history collections and research communities, such as computer graphics, product imaging, motion pictures, 3D animation, virtual and augmented realities, and education. More information and results from this project can be found at http://lightningbug.techThis award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.