Evaluation and optimization of NWB neurophysiology software and data in the cloud

项目总结-评估和优化云中的NWB软件和数据。 该提案是对U24基金的一项行政补充，名为“推进标准化进程” 通过传播NWB获得神经生理学数据。父项目的中心是为 神经数据无国界(NWB)的使用，这是神经生理学数据的数据标准，允许 神经科学研究人员将他们的数据打包并以一种易于获得和可重复使用的形式发布 其他。通过父项目，团队采取了几种方法来与用户社区接触 并降低采用NWB的准入门槛，包括主办黑客松、一对一咨询以及 教程。该团队还通过错误跟踪、测试来确保NWB代码库的持续质量 覆盖范围，并继续与科学软件开发人员接触，以帮助集成新的 用于分析、可视化、搜索和发布NWB数据集的工具。 通过我们与社区的接触，我们已经确定了优化NWB软件和 云中使用的数据是采用NWB的关键障碍，我们预计这一障碍将在 接下来的几年。随着神经生理学数据量继续快速增长，研究人员 越来越多地寻求利用云基础架构的并行处理能力来转换数据 至西北地区，并对西北地区数据进行分析。然而，目前的NWB转换工具还没有配备用于云 集成和NWB数据布局没有针对基于云的阅读和分析进行优化。要解决这些问题 为了弥补差距，我们将重点关注两个关键目标。首先，我们将评估和优化利用云资源的战略，以 使研究人员能够执行高效、经济高效的基于云的数据到NWB的转换。具体来说，我们 我将把我们的NeuroConv转换软件打包到容器中，其中包含用于 NeuroConv可以在任何云计算环境下运行，我们将开发工具来集成现有的 云资源，例如用于从云存储输入和向云存储输出转换数据。第二，我们将评估 并优化从云存储读取NWB数据，以增强基于云的分析。具体来说，我们将 将专为从云中高效读取数据而设计的Kerchunk软件包与PyNWB集成 读取NWB数据的软件，我们将评估不同数据布局策略和 优化NWB数据存储，提高基于云的访问和分析效率。 拟议工作的成功完成将建立必要的基础设施和指导 神经科学研究人员将充分利用云计算将数据转换到NWB和 NWB的数据分析。这将使研究人员能够更快地转换和分析他们的神经生理学数据 和更少的资源，这承诺改善数据共享和加快科学发现。

PROJECT SUMMARY - Evaluation and optimization of NWB software and data in the cloud. This proposal is for an administrative supplement for the U24 grant “Advancing standardization of neurophysiology data through dissemination of NWB.” The parent project centers around providing support for the usage of Neurodata Without Borders (NWB), a data standard for neurophysiology data that allows neuroscience researchers to package and publish their data in a form that is readily available and reusable by others. Through the parent project, the team is taking several approaches to engage with the user community and lower the barriers to entry for adopting NWB, including hosted hackathons, one-on-one consultations, and tutorials. The team also ensures the continued quality of the NWB codebase through bug tracking, test coverage, and continued engagement with scientific software developers to assist with the integration of new tools for analysis, visualization, search, and publication of NWB datasets. Through our engagements with the community, we have identified the need to optimize NWB software and data for usage in the cloud as a key obstacle to adoption of NWB that we anticipate will become significant in the coming years. As neurophysiology data volumes continue to grow at a rapid pace, researchers are increasingly seeking to leverage the parallel processing capabilities of cloud infrastructure for converting data to NWB and analysis of NWB data. However, the current NWB conversion tools are not yet equipped for cloud integration and the NWB data layout is not optimized for cloud-based reading and analysis. To address these gaps, we will focus on two key aims. First, we will evaluate and optimize strategies for using cloud resources to enable researchers to perform efficient, cost-effective cloud-based conversion of data to NWB. Specifically, we will package our NeuroConv conversion software into containers that contain all of the necessary elements for NeuroConv to be run on any cloud computing environment, and we will develop tools for integrating existing cloud resources, e.g., for input and output of conversion data from/to cloud storage. Second, we will evaluate and optimize reading of NWB data from cloud storage to enhance cloud-based analysis. Specifically, we will integrate the Kerchunk software package designed to read data efficiently from the cloud with the PyNWB software for reading NWB data and we will evaluate the performance of different data layout strategies and optimize the storage of NWB data to enhance the efficiency for cloud-based access and analysis. Successful completion of the proposed work will create the necessary infrastructure and guidance for neuroscience researchers to take full advantage of cloud computing for conversion of data to NWB and analysis of data in NWB. This will enable researchers to convert and analyze their neurophysiology data faster and with fewer resources, which promises to improve data sharing and expedite scientific discovery.