CIF21 DIBBs: Long Term Access to Large Scientific Data Sets: The SkyServer and Beyond

该项目旨在为更广泛的科学界创建一个围绕几个大型科学数据集建立的可持续协作生态系统。基于为斯隆数字天空测量(SDSS)SkyServer和相关项目开发的专业知识，该项目将正式确定主要系统组件，并对其进行重新设计，使其更具可重用性。该项目将完全拥有斯隆数字天空勘测档案，并将利用基于大型可扩展数据库系统的现有SDSS SkyServer框架衍生的共同、共享构建块实现的规模经济，为其持续运营提供一个强大的环境。使用这些构建块，该团队将建立和操作来自大型观测和数值模拟的开放数据档案，包括计算流体动力学、海洋环流和天体物理，达到PB级。该项目将进一步将这些工具扩展到生命科学，如大规模的下一代基因组测序实验，以及高通量神经科学成像数据。由此产生的分布式并行数据库框架将链接到用户创建的小型数据集，这些数据集也可相互协作使用，并与大型数据集结合使用。该项目将与选定的社区合作，使用我们的构建块帮助部署和提供数据，展示可移植性、通用性和规模经济；将帮助和鼓励其他机构和社区使用这些工具，同时寻求导致颠覆性变化的合作，并将建立工具，加快部署新服务和应用程序的时间表，并快速测试新想法。该项目将使个人用户能够带来他们的“小数据”，并在大数据的背景下进行协作分析。我们的具体目标是：(I)获得SDSS档案(数据库和平面文件)的完全所有权，并确保为其持续运作提供一个可扩展和强大的环境；(Ii)通过将SDSS的组件重新设计为可移植和通用的构建块，在我们关于SDSS及其特别衍生产品的十年努力的基础上再接再厉；(Iii)系统地解决因使用面向服务的体系结构(SOA)而产生的管理问题，以及由此产生的服务生命周期；(Iv)与其他科学领域的项目合作，利用我们的构建块来帮助部署和提供数据，展示可移植性、通用性和规模经济；(V)为我们的数据库集群开发可扩展的扩展，以处理高达数PB的大型数值模拟，并将其转变为开放式数值实验室；(Vi)利用我们的CasJobs协作环境来解决科学“长尾”中的小而复杂的数据问题。

The Project aims to create a sustainable collaborative ecosystem built around several large scientific data sets for the broader science community. Based upon the expertise developed for the Sloan Digital Sky Survey (SDSS) SkyServer and the associated projects the Project will formalize the main system components and reengineer them to be much more reusable. The Project will take full ownership of the Sloan Digital Sky Survey archive and will provide a robust environment for its continued operations, using an economy of scale enabled by common, shared building blocks derived from the existing SDSS SkyServer framework, based upon a large, scalable database system.Using these building blocks, the team will build and operate open data archives from large observations and numerical simulations, including computational fluid dynamics, ocean circulation and astrophysics, reaching PB scales. The Project will further extend the tools to life sciences, like large-scale, next-generation genome sequencing experiments, as well as high-throughput neuroscience imaging data. The resulting distributed, parallel database framework will be linked to small, user-created data sets that can be used also collaboratively, in conjunction with each other and the large data collections. The Project will work with selected communities to help deploying and serving data using our building blocks, demonstrating portability, generality and economies of scale; will help and encourage other institutions and communities to use the tools, while seeking collaborations that result in disruptive changes, and will build tools that accelerate the timescale to deploy new services and applications and rapidly test new ideas.The Project will enable individual users to bring their "small data" and analyze it collaboratively in the context of the large data. Our particular goals are: (i) Take full ownership of the SDSS Archive (database and flat files) and ensure a scalable and robust environment for its continued operation; (ii) Build upon our decade-long effort on SDSS and its ad-hoc spinoffs, through reengineering its components into portable and general building blocks; (iii) Systematically address curation issues arising from using a service-oriented architecture (SOA), and the resulting service life-cycle; (iv) Work with projects from additional scientific domains to help deploying and serving data using our building blocks, demonstrating portability, generality and economies of scale; (v) Develop scalable extensions to our database cluster in order to deal with large numerical simulations scaling up to petabytes, and turn them into open numerical laboratories; (vi) Use our CasJobs Collaborative Environment to address the problem of small but complex data in the "Long Tail" of science.