EarthCube IA: Collaborative Proposal: Optimal Data Layout for Scalable Geophysical Analysis in a Data-intensive Environment

遥感、卫星成像和计算技术的稳步发展，使科学家能够研究前所未有的分辨率和复杂性的地球物理现象。天基卫星或陆基雷达和辐射计设施产生的对地观测数据通常是时变的和多变量的，保存和处理可能需要万亿字节甚至千万亿字节的空间。常见的做法是从多个数据归档服务器选择数据子集并将其传输到本地计算机，然后执行数据分析任务。然而，随着观测数据量的指数增长，这种方法变得越来越不可持续。科学家可以获得详细的观测数据，但缺乏合适的、可扩展的分析能力来研究全面的数据，这成为一个日益严峻的问题。该团队将密切合作，开发、评估和部署计算机基础设施，以提高用于科学发现和教育的地球物理分析的性能和可扩展性。通过将该系统提供给其他研究人员，它将促进新的可扩展解决方案的开发。交互式地球科学应用将作为一种有效的手段来提高学生对科学和工程研究的兴趣，并为地球科学社区的发展吸引和留住学生。本研究开发了在数据密集型环境下支持可扩展地球物理分析的新技术。我们技术方法的创新之处和基础是开发了一种优化的数据布局算法，用于跨集群的分布式设备索引和放置海量异质观测数据。新的数据布局是根据对地观测数据的时空特征量身定做的，可以直接考虑包括非易失性存储资源和基于GPU和Manycore的计算节点在内的先进计算技术，并支持对大规模数据的高吞吐量和高分辨率探索。长期目标是研究理论和技术，使地球科学界能够进行可扩展的数据管理和分析。

Steady advance in remote sensing, satellite imaging, and computing technology has enabled scientists to study geophysical phenomena of unprecedented resolutions and complexity. Earth observation data generated from space-based satellites or ground-based radar and radiometer facilities are typically time-varying, and multivariate, and can take tera- or even peta-bytes of space to preserve and process. The common practice is to choose and transfer subsets of data from multiple data archive servers to local machines and then conduct data analysis tasks. However, this approach becomes increasingly unsustainable with an exponential growth of observation data size. It becomes an increasing severe problem that scientists can gain detailed observation data but lack suitable and scalable analysis capabilities to study the full extent of data. The team will work closely to develop, evaluate,and deploy the computer infrastructure to improve the performance and scalability of geophysical analysis for scientific discovery and education. By making the system available to other researchers, it will facilitate the development of new scalable solutions.Interactive geosciences applications will be used as an effective means to promote students interest in science and engineering studies, and to attract and retain students for geosciences community growth.This research develops new techniques in support of scalable geophysical analysis in a data-intensive environment. The innovation and the basis of our technique approach are to develop an optimal data layout algorithm for indexing and placing massive heterogeneous observation data across distributed devices of a cluster. The new data layout is tailored to the spatial-temporal characteristics of Earth observation data, and can directly account for advanced compute techniques, including non-volatile storage resources and GPU- and Manycore-based computing nodes, and support high-throughput and high-resolution exploration of large-scale data. The long-term goal is to study theory and technology that enable scalable data management and analysis for the geosciences community.