Interdisciplinary Scientific Data Management

科学家分析大量复杂数据的能力越来越受到限制。这些数据集不仅由仪器生成，而且由计算实验生成；最大的数值模拟的大小与仪器收集的数据相当，今年超过了pb的阈值。当科学家将他们的实验与参考模拟进行比较时，大型合成数据集的重要性变得越来越重要。所有学科都需要新的？数据仪器？它不仅可以处理大数据集，还可以处理大数据集和不同数据集的外积。虽然最大的数据集吸引了大部分公众的注意力，但它们只代表了冰山一角。经常被忽略的是，科学数据集具有幂律分布。一端是数百名科学家多年来合作收集的大量数据。这些项目通常具有一致的数据管理计划和组织，以确保广泛的社区可以访问数据产品。然而，数据的长期管理仍然是一个未解决的问题。在分布的另一端，在“长尾”中，是大量的小数据集，例如在实验室和实地研究中收集的图像、电子表格和表格。虽然单个文件很小，但它们的数量累积起来；事实上，这些小项目中汇总的数据与最大的集合中汇总的数据一样多。另一方面，这些数据集通常不像它们的大型对应物那样有很好的文档记录。对于大多数科学家来说，成为数据管理专家并花时间记录数据以供以后重用是没有什么回报的。事实上，人工清理数据集的过程被称为大数据的条带挖掘：这是一种丑陋的、资源密集的工作，会留下很大的伤疤。约翰霍普金斯大学的科学家们已经建立了创新的框架来发布从天文学到湍流和环境科学等广泛学科的科学数据。这些项目已经共享了一些用于数据管理的公共组件。本项目将把更多现有的独立组件连接成一个连贯的组件，探索如何扩展数据服务以应对数据分布的“长尾”，并展示跨学科基础数据管理任务中的重叠。该项目有四个部分：(i)继续并加强斯隆数字巡天的工作；（ii）将大型数值模拟转变为易于使用的数值实验室；（iii）增强现有的环境传感器端到端系统，并将其与其他现场数据集成；（iv）增强和推广一套核心协作工具，并应用这些工具来帮助应对科学数据“长尾”的挑战。这些项目包括斯隆数字巡天（SDSS）——世界上最常用的天文设施——及其CASJOBs/MyDB协作环境。该框架将扩展到其他科学领域，如现场环境监测和野外生物学。这将通过整合来自巴尔的摩生态系统研究项目的土壤生态学数据和通过无线传感器网络自动收集的数据来证明。该项目还将测试，如何使用一个简单的，类似于“DropBox”的界面（即在线存储和共享）来克服一些阻碍科学家发布大量增值数据的障碍。最后，该项目将探索如何将更小和更大的数值模拟放入交互式的、可公开访问的数值实验室，使用当前来自湍流和天文学的数据集。这些资金将支持人们：数据科学家、数据库管理员、博士后研究员、学生和程序员的组合，共同努力“连接点”，并将额外的数据集上线。该项目将增强几个公开可用数据集的公共接口，构建一个易于使用的环境原型，将小型用户数据上传到协作环境中，并为环境科学领域的一个新的公民科学项目创建一个框架。

Scientists are increasingly limited by their ability to analyze the large amounts of complex data available. These data sets are generated not only by instruments but also computational experiments; the sizes of the largest numerical simulations are on par with data collected by instruments, crossing the petabyte threshold this year. The importance of large synthetic data sets is increasingly important, as scientists compare their experiments to reference simulations. All disciplines need a new ?instrument for data? that can deal not only with large data sets but the cross product of large and diverse data sets. While the largest data sets have captured most of the public attention, they only represent the tip of the iceberg. What is often missed is that scientific data sets have a power law distribution. At one end are the very large data collections compiled by hundreds of scientists collaborating over multiple years. These projects typically have coherent data management plans and organization to ensure that the data products are accessible to a wide community. Nevertheless, the long-term curation of the data is still an unsolved problem. At the other end of the distribution, in the "long tail", are the very large numbers of small data sets, such as the images, spreadsheets and tables collected in laboratories and field studies. While the individual files are small, their numbers add up; in fact, there is as much data aggregated in these small items as in the biggest collections. On the other hand, these data sets are often not as well documented as their bigger counterparts. For most scientists there is little reward in becoming a data management expert and devoting the time required to documenting the data for later reuse. In fact, the process of manually cleaning data sets has been called the strip mining of big data: an ugly and resource intensive effort that leaves big scars.Scientists at the Johns Hopkins University have built innovative frameworks to publish scientific data across a wide range of disciplines, from astronomy to turbulence, and environmental science. These projects already share some common components for data management. This project will connect more of the existing independent components into a coherent one, explore how to scale the data services to deal with the "long tail" of the data distribution, and demonstrate the overlap in the basic data management tasks across disciplines. The project has four parts: (i) continue and enhance the efforts on the Sloan Digital Sky Survey, (ii) turn large numerical simulations into easy-to-use numerical laboratories, (iii) enhance an existing end-to-end system for environmental sensors and integrate it with other field data, (iv) enhance and generalize a set of core collaborative tools, and apply these to help with the challenge of the "long tail" of scientific data. The projects involve the Sloan Digital Sky Survey (SDSS) -- the world's most used astronomy facility -- and its CASJOBs/MyDB collaborative environment The framework will be extended to other areas of science, like in-situ environmental monitoring and field biology. This will be demonstrated by integrating data in soil ecology from the Baltimore Ecosystem Study project with data collected automatically, via a wireless sensor network. The project will also test, how a simple, "DropBox"-like interface (i.e., online storage and sharing) can be used to overcome some of the barriers that prevent scientists from publishing much of their value-added data. Finally, the project will explore how smaller and larger numerical simulations can be placed into interactive, publicly accessible numerical laboratories, using data sets currently from turbulence and astronomy. The funds will support people: a combination of data scientists, database administrators, postdoctoral fellows, students and programmers working together to "connect the dots" and bring additional data sets on line. The project will enhance the public interfaces of several publicly available data sets, prototype an easy-to-use environment to upload small user data into a collaborative environment, and create a framework for a new citizen-science project in environmental science.