CIF21 DIBBs: Long Term Access to Large Scientific Data Sets: The SkyServer and Beyond

该项目旨在围绕多个大型科学数据集为更广泛的科学界创建一个可持续的协作生态系统。基于为斯隆数字巡天 (SDSS) SkyServer 和相关项目开发的专业知识，该项目将正式确定主要系统组件并重新设计它们，使其更加可重用。该项目将完全拥有斯隆数字巡天档案，并为其持续运营提供一个强大的环境，使用由现有 SDSS SkyServer 框架衍生的通用、共享构建块实现的规模经济，基于大型、可扩展的数据库系统。使用这些构建块，该团队将根据大型观测和数值模拟（包括计算流体动力学、海洋环流和天体物理学）构建和操作开放数据档案，达到 PB 级。该项目将进一步将这些工具扩展到生命科学，例如大规模的下一代基因组测序实验以及高通量神经科学成像数据。由此产生的分布式并行数据库框架将链接到用户创建的小型数据集，这些数据集也可以与彼此以及大型数据集合一起协作使用。该项目将与选定的社区合作，帮助使用我们的构建块部署和提供数据，展示可移植性、通用性和规模经济；将帮助和鼓励其他机构和社区使用这些工具，同时寻求带来颠覆性变革的合作，并将构建加快部署新服务和应用程序并快速测试新想法的工具。该项目将使个人用户能够携带他们的“小数据”并在大数据的背景下对其进行协作分析。我们的具体目标是： (i) 完全拥有 SDSS Archive（数据库和平面文件），并确保为其持续运行提供可扩展且强大的环境； (ii) 通过将其组件重新设计为可移植和通用的构建块，以我们在 SDSS 及其临时衍生产品方面长达十年的努力为基础； (iii) 系统地解决因使用面向服务的架构（SOA）而产生的管理问题以及由此产生的服务生命周期； (iv) 与其他科学领域的项目合作，帮助使用我们的构建模块部署和提供数据，展示可移植性、通用性和规模经济； (v) 开发数据库集群的可扩展扩展，以处理高达 PB 级的大型数值模拟，并将其转变为开放数值实验室； (vi) 使用我们的 CasJobs 协作环境来解决科学“长尾”中小而复杂的数据问题。

The Project aims to create a sustainable collaborative ecosystem built around several large scientific data sets for the broader science community. Based upon the expertise developed for the Sloan Digital Sky Survey (SDSS) SkyServer and the associated projects the Project will formalize the main system components and reengineer them to be much more reusable. The Project will take full ownership of the Sloan Digital Sky Survey archive and will provide a robust environment for its continued operations, using an economy of scale enabled by common, shared building blocks derived from the existing SDSS SkyServer framework, based upon a large, scalable database system.Using these building blocks, the team will build and operate open data archives from large observations and numerical simulations, including computational fluid dynamics, ocean circulation and astrophysics, reaching PB scales. The Project will further extend the tools to life sciences, like large-scale, next-generation genome sequencing experiments, as well as high-throughput neuroscience imaging data. The resulting distributed, parallel database framework will be linked to small, user-created data sets that can be used also collaboratively, in conjunction with each other and the large data collections. The Project will work with selected communities to help deploying and serving data using our building blocks, demonstrating portability, generality and economies of scale; will help and encourage other institutions and communities to use the tools, while seeking collaborations that result in disruptive changes, and will build tools that accelerate the timescale to deploy new services and applications and rapidly test new ideas.The Project will enable individual users to bring their "small data" and analyze it collaboratively in the context of the large data. Our particular goals are: (i) Take full ownership of the SDSS Archive (database and flat files) and ensure a scalable and robust environment for its continued operation; (ii) Build upon our decade-long effort on SDSS and its ad-hoc spinoffs, through reengineering its components into portable and general building blocks; (iii) Systematically address curation issues arising from using a service-oriented architecture (SOA), and the resulting service life-cycle; (iv) Work with projects from additional scientific domains to help deploying and serving data using our building blocks, demonstrating portability, generality and economies of scale; (v) Develop scalable extensions to our database cluster in order to deal with large numerical simulations scaling up to petabytes, and turn them into open numerical laboratories; (vi) Use our CasJobs Collaborative Environment to address the problem of small but complex data in the "Long Tail" of science.