Boss: A cloud-based data archive for electron microscopy and x-ray microtomography

Boss：用于电子显微镜和 X 射线显微断层扫描的基于云的数据存档

• 批准号: 9769873
• 负责人: BROCK A. WESTER
• 金额: $ 63.76万
• 依托单位: JOHNS HOPKINS UNIVERSITY
• 依托单位国家: 美国
• 财政年份: 2018
• 资助国家: 美国
• 起止时间: 2018-08-24 至 2023-05-31
• 项目状态: 已结题
• 来源: https://reporter.nih.gov/project-details/9769873
• 关键词: 3-Dimensional; Adoption; Architecture; Archives; BRAIN initiative; Brain; Brain region; Client; Collaborations; Collection; Communities; Complex; Computer software; Cryopreservation; Data; Data Analytics; Data Collection; Data Quality; Data Set; Data Storage and Retrieval; Databases; Development; Docking; Electron Microscopy; Engineering; Ensure; Equilibrium; Fostering; Generations; Geography; Human; Image; Imagery; Imaging Techniques; Individual; Ingestion; Laboratories; Manuals; Maps; Memory; Metadata; Methods; Modality; Mus; Neurons; Neurosciences; Online Systems; Performance; Play; Preparation; Process; Production; Property; Pythons; Reproducibility; Research; Research Personnel; Resolution; Roentgen Rays; Role; Running; Sampling; Secure; Services; Source; Standardization; Structure; System; Three-Dimensional Image; Time; Tissues; Training; United States National Institutes of Health; Validation; Visualization software; Work; abstracting; application programming interface; brain volume; cloud based; computerized data processing; computing resources; cost; cost efficient; data access; data archive; data sharing; design; experience; experimental study; file format; flexibility; gigabyte; high resolution imaging; improved; indexing; instrument; interest; member; millimeter; multimodality; neuroimaging; novel; petabyte; search engine; software development; spatial memory; submicron; terabyte; tool; web services; web site

Project Abstract Due to recent technological advances, it is possible to image the high-resolution structure of brain volumes at spatial extents that are much larger than was previously possible. Emerging X-ray microtomography (XRM) methods allow for the collection of whole mouse brains in a high-throughput paradigm, permitting the generation of sub-micron three-dimensional image volumes in less than a day without the alignment challenges or tissue clearing approaches of other methods. Similarly, electron microscopy (EM) efforts now routinely exceed 100 terabytes in scale and projects are underway to map cubic millimeters of brain regions, resulting in petabytes of image and annotation data. Both of these methods are widely used throughout the BRAIN Initiative and the broader neuroscience community, and the instruments required to collect these datasets are becoming more common and higher throughput resulting in an increased need for data storage and archival solutions that can accommodate these larger datasets that are being generated at an increasing pace. Finally, the sample preparation of XRM and EM are compatible with and amenable to co-registration, and work is underway to pursue multimodal experiments; new instruments are now available with the ability to perform both XRM and EM data collection from a single sample. Existing paradigms for data storage and access are often insufficient to accommodate the required storage, processing, and dissemination needed to fully exploit the generated data. At this scale, traditional analysis approaches are often ineffective; for example, it is difficult for a human to view all of data collected or manually annotate more than a small fraction of the volume. Contemporary analysis approaches leveraging automated methods require robust and efficient access to data, which can be challenging when managing massive datasets spread across many files. Without a standard data storage mechanism, data access is cumbersome, storage is expensive and can lack sufficient durability, metadata is unreliable or unavailable and may not be attributable in useful ways, and file formats and organization are often different across laboratories, resulting in a high-barrier for collaboration and sharing. Thus, we propose the Block and Object Storage Service Database (bossDB) to deliver a high-performance, cost efficient data archive by utilizing a cloud-based tiered storage architecture, where data is seamlessly migrated between low cost, durable object storage (i.e., S3) and a fast in-memory spatial data store. This system will be developed through an agile process that will actively fold in community stakeholders for regular reviews and continuous opportunities for design input, and will provide and support integration of a robust suite of user-facing tools that are vital to foster community adoption, such as a web-based management console and visualization tool, a Python SDK for programmatic access, and a client to facilitate large-scale ingest of data into the platform. We will build an integrated, managed framework that will enable compute data quality metrics on large datasets, and a metadata store to capture experimental details, dataset properties and information about available results. Our approach provides a secure, versioned API to facilitate programmatic access to data through a standardized and stable interface. For members of the community who prefer a locally-deployed solution, we will additionally ensure bossDB data storage capabilities exist in a local (on-premises) deployable version of the archive, and will integrate with the other major community solution, DVID, which provides complementary capabilities. This proposal will result in a professionally-engineered, highly-available data archive that provides solutions to many of the barriers associated with large-scale neuroscience discovery. Through a service-oriented architecture, our approach is flexible and designed to provide many capabilities to the user while abstracting most of the underlying technical details so that neuroscientists and data analytics researchers can focus on the scientific questions of greatest interest. We believe that providing this archive will enable many new experiments in XRM, EM and mutlimodal approaches, and can be adapted as user and community needs evolve.

项目摘要 由于最近的技术进步，有可能在1000米处对脑体积的高分辨率结构进行成像。 空间范围比以前可能的要大得多。新兴X射线显微断层成像（XRM） 方法允许以高通量范例收集整个小鼠脑， 在不到一天的时间内生成亚微米三维图像体积， 挑战或其他方法的组织清除方法。同样，电子显微镜（EM）的努力，现在 通常超过100太字节的规模和项目正在进行中，以绘制立方毫米的大脑区域， 从而产生PB级的图像和注释数据。这两种方法都广泛应用于 BRAIN Initiative和更广泛的神经科学界，以及收集这些信息所需的工具。 数据集变得越来越普遍，吞吐量也越来越高，这导致对数据存储的需求不断增加 和归档解决方案，这些解决方案可以容纳这些以越来越高的速度生成的大型数据集， 步伐。最后，XRM和EM的样品制备与共配准相容且易于共配准， 目前正在进行多模式实验;新的仪器现在可以使用， 从单个样品中进行XRM和EM数据收集。 现有的数据存储和访问范式通常不足以容纳所需的存储， 处理和传播，以充分利用所产生的数据。在这种规模下，传统的分析 方法通常是无效的;例如，人类很难查看收集的所有数据， 手动注释体积的一小部分以上。利用现代分析方法 自动化方法需要强大而高效的数据访问，这在管理 大量数据集分布在许多文件中。如果没有标准的数据存储机制， 笨重、存储昂贵且缺乏足够的持久性，元数据不可靠或不可用， 可能无法以有用的方式归因，并且文件格式和组织通常在不同的 实验室，导致合作和共享的高壁垒。 因此，我们提出了块和对象存储服务数据库（bossDB），以提供高性能， 通过利用基于云的分层存储体系结构实现经济高效的数据归档， 在低成本、持久的对象存储（即，S3）和快速内存空间数据存储。这 系统将通过敏捷流程开发，该流程将积极融入社区利益相关者， 定期审查和设计输入的持续机会，并将提供和支持 强大的面向用户的工具套件，对于促进社区采用至关重要，例如基于Web的 管理控制台和可视化工具，用于编程访问的Python SDK，以及用于 将大规模数据导入平台。我们将建立一个综合管理的框架， 计算大型数据集上的数据质量指标，以及元数据存储以捕获实验细节、数据集 属性和有关可用结果的信息。我们的方法提供了一个安全的、版本化的API， 通过标准化和稳定的界面，促进对数据的编程访问。成员的 对于更喜欢本地部署解决方案的社区，我们还将确保bossDB数据存储 归档的本地（本地）可部署版本中存在功能，并将与其他版本集成 主要的社区解决方案，DVID，它提供了补充功能。这一提议将导致 经过专业设计的高可用性数据归档，可为许多障碍提供解决方案 与大规模神经科学发现有关。通过面向服务的体系结构，我们的方法是 灵活，旨在为用户提供许多功能，同时抽象大多数底层 技术细节，以便神经科学家和数据分析研究人员可以专注于以下科学问题： 最大的兴趣。我们相信，提供这个档案将使许多新的实验，在XRM，EM和 多模式方法，并可随着用户和社区需求的变化而调整。