CC*Data: National Cyberinfrastructure for Scientific Data Analysis at Scale (SciDAS)

科学发现越来越依赖于需要以前所未有的规模进行计算的海量数据集。实验室的计算机和电子表格根本无法处理从DNA测序仪等现代测量设备流出的数据。科学实验现在需要了解基础科学和网络基础设施(CI)生态系统，以设计和执行必要的计算。幸运的是，来自公共和私营部门的重大战略支持正在国家一级创建一个分布式计算生态系统，以帮助满足大型数据集的计算需求。该项目名为“大规模科学数据分析”，旨在提高获取国家资源的灵活性和可及性，帮助研究人员更有效地利用更广泛的资源。本系统是利用大规模系统生物学和水文学数据集开发的，但可以扩展到许多其他领域。在技术层面上，本系统联合访问多个国家CI资源，包括NSF云、开放科学网格、极端科学和工程发现环境(XSEDE v2.0)、千万亿级超级计算机(如COMET)和校园资源。本科学数据仓库系统的核心是使用ExoGENI动态联网基础设施，以实现这些资源和数据储存库之间的第二层连接和数据移动。SciDAS依靠集成的面向规则的数据系统(IRODS)，增强了软件定义的网络(SDN)功能，以支持网络感知数据管理决策和网络资源的高效使用。利用数据共享和计算基础设施的分布式和可扩展特性，针对计算机和数据局部性进行优化，从而提高工作流程的性能和科学生产力。系统生物学和水文学方面的科学发现用例将推动千万亿级网络基础设施的发展，同时为领域科学家产生有用的成果。

Scientific discovery is increasingly dependent on huge datasets that require computing at unprecedented scale. Laboratory computers and spreadsheets simply cannot handle the data flowing from modern measurement devices, such as DNA sequencers. Scientific experiments now require understanding of both the underlying science and the cyberinfrastructure (CI) ecosystem to design and execute necessary computations. Fortunately, significant and strategic support from the public and private sectors is creating a distributed computational ecosystem at the national level to help meet the computational demands of large datasets. This project, the Scientific Data Analysis at Scale (SciDAS) is designed to improve flexibility and accessibility to national resources, helping researchers more effectively use a broader array of these resources. SciDAS is developed using large-scale systems biology and hydrology datasets, but is extensible to many other domains.On a technical level, SciDAS federates access to multiple national CI resources including NSF Cloud, Open Science Grid, the Extreme Science and Engineering Discovery Environment (XSEDE v2.0), petascale supercomputers such as COMET, and campus resources. Central to SciDAS is the use of ExoGENI dynamic networked infrastructure to enable Layer-2 connectivity and data movement between these resources and data repositories. SciDAS relies on the integrated-Rule-Oriented-Data-System (iRODS), enhanced with software-defined-networking (SDN) capabilities, to support network-aware data management decisions and efficient use of network resources. The distributed and scalable nature of both the data-sharing and the compute infrastructure are exploited to optimize for computer and data locality, boosting the performance of workflows and scientific productivity. Scientific discovery use cases in systems biology and hydrology will drive cyberinfrastructure development at the petascale level while simultaneously generating useful results for domain scientists.