Interdisciplinary Scientific Data Management

科学家分析现有的大量复杂数据的能力越来越有限。这些数据集不仅是由仪器产生的，也是由计算实验产生的；最大的数值模拟的大小与仪器收集的数据相当，今年超过了PB的门槛。随着科学家将他们的实验与参考模拟进行比较，大型合成数据集的重要性变得越来越重要。所有学科都需要一种新的数据工具。这不仅可以处理大型数据集，而且可以处理大型和多样化数据集的叉积。虽然最大的数据集吸引了公众的大部分注意力，但它们只代表了冰山一角。人们经常忽略的是，科学数据集具有幂定律分布。一端是由数百名科学家多年合作汇编的非常庞大的数据集合。这些项目通常有连贯的数据管理计划和组织，以确保数据产品可供广大社区访问。然而，数据的长期管理仍然是一个悬而未决的问题。在分布的另一端，在“长尾”中，是大量的小数据集，如在实验室和实地研究中收集的图像、电子表格和表格。虽然单个文件很小，但它们的数量会累积起来；事实上，这些小项目中聚合的数据与最大的集合中的数据一样多。另一方面，这些数据集通常没有更大的数据集那么好地记录下来。对于大多数科学家来说，成为一名数据管理专家，并将所需的时间投入到记录数据以供以后重复使用，几乎没有什么回报。事实上，手动清理数据集的过程被称为大数据的条带式挖掘：这是一种丑陋的、资源密集型的工作，会留下巨大的伤疤。约翰·霍普金斯大学的科学家们建立了创新的框架，发布从天文学到湍流和环境科学等广泛学科的科学数据。这些项目已经共享了一些用于数据管理的通用组件。该项目将把更多现有的独立组件连接成一个连贯的组件，探索如何扩展数据服务以处理数据分布的“长尾”，并展示跨学科的基本数据管理任务的重叠。该项目包括四个部分：(I)继续和加强斯隆数字天空调查的努力；(Ii)将大型数值模拟变成易于使用的数值实验室；(Iii)加强现有的环境传感器端到端系统，并将其与其他实地数据相结合；(Iv)增强和推广一套核心协作工具，并将其应用于帮助应对科学数据“长尾”的挑战。这些项目涉及斯隆数字巡天(SDSS)--世界上使用最多的天文设施--及其CASJOBS/MyDB协作环境。该框架将扩展到其他科学领域，如现场环境监测和野外生物学。这将通过将巴尔的摩生态系统研究项目的土壤生态学数据与通过无线传感器网络自动收集的数据相结合来进行演示。该项目还将测试如何使用一个简单的、类似Dropbox的界面(即在线存储和共享)来克服阻止科学家发布大量增值数据的一些障碍。最后，该项目将探索如何利用目前来自湍流和天文学的数据集，将越来越小的、更大的数值模拟放入交互式的、可公开访问的数值实验室。这笔资金将支持人们：数据科学家、数据库管理员、博士后研究员、学生和程序员共同努力，将更多的数据集连接起来。该项目将增强几个公开可用的数据集的公共接口，建立一个易于使用的环境原型，将小型用户数据上传到一个协作环境中，并为一个新的环境科学公民科学项目创建一个框架。

Scientists are increasingly limited by their ability to analyze the large amounts of complex data available. These data sets are generated not only by instruments but also computational experiments; the sizes of the largest numerical simulations are on par with data collected by instruments, crossing the petabyte threshold this year. The importance of large synthetic data sets is increasingly important, as scientists compare their experiments to reference simulations. All disciplines need a new ?instrument for data? that can deal not only with large data sets but the cross product of large and diverse data sets. While the largest data sets have captured most of the public attention, they only represent the tip of the iceberg. What is often missed is that scientific data sets have a power law distribution. At one end are the very large data collections compiled by hundreds of scientists collaborating over multiple years. These projects typically have coherent data management plans and organization to ensure that the data products are accessible to a wide community. Nevertheless, the long-term curation of the data is still an unsolved problem. At the other end of the distribution, in the "long tail", are the very large numbers of small data sets, such as the images, spreadsheets and tables collected in laboratories and field studies. While the individual files are small, their numbers add up; in fact, there is as much data aggregated in these small items as in the biggest collections. On the other hand, these data sets are often not as well documented as their bigger counterparts. For most scientists there is little reward in becoming a data management expert and devoting the time required to documenting the data for later reuse. In fact, the process of manually cleaning data sets has been called the strip mining of big data: an ugly and resource intensive effort that leaves big scars.Scientists at the Johns Hopkins University have built innovative frameworks to publish scientific data across a wide range of disciplines, from astronomy to turbulence, and environmental science. These projects already share some common components for data management. This project will connect more of the existing independent components into a coherent one, explore how to scale the data services to deal with the "long tail" of the data distribution, and demonstrate the overlap in the basic data management tasks across disciplines. The project has four parts: (i) continue and enhance the efforts on the Sloan Digital Sky Survey, (ii) turn large numerical simulations into easy-to-use numerical laboratories, (iii) enhance an existing end-to-end system for environmental sensors and integrate it with other field data, (iv) enhance and generalize a set of core collaborative tools, and apply these to help with the challenge of the "long tail" of scientific data. The projects involve the Sloan Digital Sky Survey (SDSS) -- the world's most used astronomy facility -- and its CASJOBs/MyDB collaborative environment The framework will be extended to other areas of science, like in-situ environmental monitoring and field biology. This will be demonstrated by integrating data in soil ecology from the Baltimore Ecosystem Study project with data collected automatically, via a wireless sensor network. The project will also test, how a simple, "DropBox"-like interface (i.e., online storage and sharing) can be used to overcome some of the barriers that prevent scientists from publishing much of their value-added data. Finally, the project will explore how smaller and larger numerical simulations can be placed into interactive, publicly accessible numerical laboratories, using data sets currently from turbulence and astronomy. The funds will support people: a combination of data scientists, database administrators, postdoctoral fellows, students and programmers working together to "connect the dots" and bring additional data sets on line. The project will enhance the public interfaces of several publicly available data sets, prototype an easy-to-use environment to upload small user data into a collaborative environment, and create a framework for a new citizen-science project in environmental science.