Elements: PASSPP: Provenance-Aware Scalable Seismic Data Processing with Portability

我们对地球内部深处的大部分了解来自于对地面运动数据的分析，这些数据记录了大地震产生的地震波，这些地震波来自整个星球上的仪器。地震学家已经开发出一长串方法，将现代地震数据处理成图像？地球？S腹地。我们对地球内部的了解在很大程度上受到了可用来构建这些“图像”的工具的分辨率的限制。目前，数据量的海量增加将地震学的数据处理基础设施推向了崩溃的边缘。无法处理这种规模的数据给科学发现带来了巨大的障碍，特别是对于资源有限的规模较小的研究小组。为了帮助改善这种情况，本项目引入了一种新的数据管理和处理系统，该系统具有可移植和可扩展的特点，可以运行在从个人计算机到大型超级计算机的任何平台上。通过利用和集成来自云计算和高性能计算(HPC)社区的复杂工具，该系统可以填补数据中心提供的海量数据与当前工具提供的数据管理和处理能力不足之间不断扩大的差距。社区将能够在数据中心之外无缝地发现、访问、传输和处理数据和元数据。该项目还将成为利用海量数据进行新颖研究的基础，以改变我们研究地球结构、组成和演化的方式。该项目旨在开发一个由基于数据流计算模型的可扩展并行处理框架、以文档存储为中心的NoSQL数据库系统和基于容器的虚拟化环境组成的地震数据管理与处理系统。可伸缩处理组件将基于迭代MAP-Reduce模型，使用ApacheSpark来处理不同规模系统中的数据调度和流动。通过管理在使用MongoDB处理期间创建的所有数据，包括处理生成的元数据、处理的波形数据、处理参数和日志输出，将启用来源感知数据管理。所有这些核心组件以及在不同系统上配置和部署框架的脚本都将使用Singulity进行包装，以提供可移植性。所有这些组件都服务于该项目的两个主要目标：建立一个系统，使普通地震学算法能够在现代高性能计算平台上有效运行；以及为具有一般编程经验的地震学家提供手段，以实现他们自己的算法，以扩展该系统。该系统将作为基础设施，使较小的研究小组能够进行数据密集型研究，如深度学习，这些研究小组通常没有必要的人力以可持续的方式管理和处理海量数据。通过能够处理由越来越多的仪器收集的海量数据，它将促进该领域向数据密集型科学发现范式的过渡。该奖项反映了NSF的法定使命，并通过使用基金会的智力优势和更广泛的影响审查标准进行评估，被认为值得支持。

Most of what we know about the Earth's deep interior comes from the analysis of ground motion data recording seismic waves produced by large earthquakes from instruments around the entire planet. Seismologists have developed a long list of methods to process modern seismic data to ?image? the Earth?s interior. Much of our understanding of Earth's interior has been limited by the resolution of the tools available to construct these "images". At present, the massive increase in data volume has pushed the data processing infrastructure of seismology to the breaking point. The inability to handle data of this scale has imposed significant barrier to scientific discoveries, especially for the smaller research groups with limited resources. Aiming to help improve this situation, this project introduces a new data management and processing system that is portable and scalable to run on any platforms from a personal computer to a large-scale supercomputer. By leveraging and integrating sophisticated tools from cloud computing and high-performance computing (HPC) communities, the system can fill in the widening gap between the massive data made available by data centers and the inadequacy of data management and processing capability provided with current tools. Seamless discovery, access, transfer, and processing of data and metadata outside of data centers will become possible for the community. This project will also serve as the foundation to enable novel research utilizing massive data to change the way we study the structure, composition, and evolution of the Earth. This project aims to develop a seismic data management and processing system that is composed of a scalable parallel processing framework based on dataflow computation model, a NoSQL database system centered on document store, and a container-based virtualization environment. The scalable processing component will be based on the iterative map-reduce model using Apache Spark to handle scheduling and flow of data through systems of different scales. The provenance-aware data management will be enabled by managing all data created during processing with MongoDB, including process generated metadata, processed waveform data, processing parameters, and the log outputs. All these core components as well as a script to configure and deploy the framework on different systems will be containerized with Singularity to provide portability. All these components serve the two primary goals of the project: produce a system that will allow common seismology algorithms to run effectively on modern HPC platforms; and provide the means for seismologists with average experience in programming to implement their own algorithms to extend the system. The system will serve as the infrastructure to make data intensive research such as deep learning possible for smaller research groups that usually don't have the necessary manpower to manage and process massive data in a sustainable fashion. By enabling the ability to process massive data collected by increasing number of instruments, it will facilitate the transition of the field into data-intensive paradigm of science discovery.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.