Interdisciplinary Scientific Data Management

科学家分析大量复杂数据的能力越来越受到限制。这些数据集不仅是由仪器产生的，也是由计算实验产生的；最大数值模拟的规模与仪器收集的数据相当，今年突破了拍字节的门槛。随着科学家将他们的实验与参考模拟进行比较，大型合成数据集的重要性变得越来越重要。所有学科都需要一种新的“数据工具”？它不仅可以处理大型数据集，还可以处理大型且多样化的数据集的叉积。虽然最大的数据集吸引了大部分公众的注意力，但它们仅代表冰山一角。人们经常忽略的是，科学数据集具有幂律分布。一方面是由数百名科学家多年来合作收集的非常大的数据集。 这些项目通常具有连贯的数据管理计划和组织，以确保广泛的社区可以访问数据产品。尽管如此，数据的长期管理仍然是一个未解决的问题。分布的另一端，即“长尾”，是大量的小数据集，例如在实验室和现场研究中收集的图像、电子表格和表格。虽然单个文件很小，但它们的数量加起来；事实上，这些小项目中聚合的数据与最大集合中聚合的数据一样多。另一方面，这些数据集通常不如其较大的对应数据集那样有详细记录。对于大多数科学家来说，成为数据管理专家并投入时间记录数据以供以后重用并没有什么回报。事实上，手动清理数据集的过程被称为大数据的条带挖掘：这是一种丑陋且资源密集的工作，会留下巨大的伤痕。约翰·霍普金斯大学的科学家们建立了创新框架来发布从天文学到湍流和环境科学等广泛学科的科学数据。这些项目已经共享一些通用的数据管理组件。该项目将把更多现有的独立组件连接成一个连贯的组件，探索如何扩展数据服务以处理数据分布的“长尾”，并展示跨学科的基本数据管理任务的重叠。该项目分为四个部分：（i）继续并加强斯隆数字巡天工作，（ii）将大型数值模拟转变为易于使用的数值实验室，（iii）增强现有的环境传感器端到端系统并将其与其他现场数据集成，（iv）增强和推广一套核心协作工具，并应用这些工具来帮助应对科学数据“长尾”的挑战。这些项目涉及斯隆数字巡天 (SDSS)——世界上使用最广泛的天文学设施——及其 CASJOBs/MyDB 协作环境。该框架将扩展到其他科学领域，如现场环境监测和野外生物学。这将通过将巴尔的摩生态系统研究项目的土壤生态数据与通过无线传感器网络自动收集的数据相结合来证明。该项目还将测试如何使用一个简单的、类似“DropBox”的界面（即在线存储和共享）来克服一些阻碍科学家发布大部分增值数据的障碍。最后，该项目将探索如何使用当前来自湍流和天文学的数据集将更小和更大的数值模拟放入交互式、可公开访问的数值实验室中。 这些资金将支持人们：数据科学家、数据库管理员、博士后研究员、学生和程序员共同努力“连接点”并将更多数据集上线。该项目将增强多个公开数据集的公共接口，构建一个易于使用的环境原型，以将小型用户数据上传到协作环境中，并为环境科学领域的新公民科学项目创建一个框架。

Scientists are increasingly limited by their ability to analyze the large amounts of complex data available. These data sets are generated not only by instruments but also computational experiments; the sizes of the largest numerical simulations are on par with data collected by instruments, crossing the petabyte threshold this year. The importance of large synthetic data sets is increasingly important, as scientists compare their experiments to reference simulations. All disciplines need a new ?instrument for data? that can deal not only with large data sets but the cross product of large and diverse data sets. While the largest data sets have captured most of the public attention, they only represent the tip of the iceberg. What is often missed is that scientific data sets have a power law distribution. At one end are the very large data collections compiled by hundreds of scientists collaborating over multiple years. These projects typically have coherent data management plans and organization to ensure that the data products are accessible to a wide community. Nevertheless, the long-term curation of the data is still an unsolved problem. At the other end of the distribution, in the "long tail", are the very large numbers of small data sets, such as the images, spreadsheets and tables collected in laboratories and field studies. While the individual files are small, their numbers add up; in fact, there is as much data aggregated in these small items as in the biggest collections. On the other hand, these data sets are often not as well documented as their bigger counterparts. For most scientists there is little reward in becoming a data management expert and devoting the time required to documenting the data for later reuse. In fact, the process of manually cleaning data sets has been called the strip mining of big data: an ugly and resource intensive effort that leaves big scars.Scientists at the Johns Hopkins University have built innovative frameworks to publish scientific data across a wide range of disciplines, from astronomy to turbulence, and environmental science. These projects already share some common components for data management. This project will connect more of the existing independent components into a coherent one, explore how to scale the data services to deal with the "long tail" of the data distribution, and demonstrate the overlap in the basic data management tasks across disciplines. The project has four parts: (i) continue and enhance the efforts on the Sloan Digital Sky Survey, (ii) turn large numerical simulations into easy-to-use numerical laboratories, (iii) enhance an existing end-to-end system for environmental sensors and integrate it with other field data, (iv) enhance and generalize a set of core collaborative tools, and apply these to help with the challenge of the "long tail" of scientific data. The projects involve the Sloan Digital Sky Survey (SDSS) -- the world's most used astronomy facility -- and its CASJOBs/MyDB collaborative environment The framework will be extended to other areas of science, like in-situ environmental monitoring and field biology. This will be demonstrated by integrating data in soil ecology from the Baltimore Ecosystem Study project with data collected automatically, via a wireless sensor network. The project will also test, how a simple, "DropBox"-like interface (i.e., online storage and sharing) can be used to overcome some of the barriers that prevent scientists from publishing much of their value-added data. Finally, the project will explore how smaller and larger numerical simulations can be placed into interactive, publicly accessible numerical laboratories, using data sets currently from turbulence and astronomy. The funds will support people: a combination of data scientists, database administrators, postdoctoral fellows, students and programmers working together to "connect the dots" and bring additional data sets on line. The project will enhance the public interfaces of several publicly available data sets, prototype an easy-to-use environment to upload small user data into a collaborative environment, and create a framework for a new citizen-science project in environmental science.