CRII: III: Declarative array processing for large-scale scientific analyses

科学家通过运行在数十万个处理核心上的数据密集型分析来了解复杂的自然现象。然而，快速并行地探索非常庞大的数据集以获得洞察力是具有挑战性的。分析大于内存的数据集暴露了存储层次结构的错综复杂，并需要基于预期的数据量和系统体系结构进行不同的实施。因此，使用大规模计算设施的领域科学家面临两难境地：他们需要根据不断发展的系统基础设施永久维护和调整他们的数据处理代码，或者将他们的调查限制在可以在合理时间内完成的分析，因为数据集的大小不断增长。声明性数据处理技术可以减轻科学家管理如何访问或存储科学数据的负担。尽管科学家经常使用许多声明式数据管理系统，但在回答任何科学查询之前，这些系统需要进行耗时的数据转换，如加载、分块和重新分区。此外，许多数据管理系统承担了对底层硬件的完全控制，并且没有注意到通过大规模计算设施的批处理执行范例提供的优化和扩展机会。为了解决这一研究差距，我们将研究数据管理和科学计算交叉领域的技术，如何分配资源，以及如何主动管理并行I/O和分布式内存。为了影响科学实践，我们将开发一个原型运行时系统，该系统将扩大已建立的具有声明性查询功能的科学文件格式库，用于领导力计算设施中的数据分析。有关更多信息，请参阅项目网站：http://go.osu.edu/insitu_analysis

Scientists understand complex natural phenomena through data-intensive analyses that run on hundreds of thousands of processing cores. Quickly exploring very large datasets in parallel for insights, however, is challenging. Analyzing larger-than-memory datasets exposes the intricacies of the storage hierarchy and necessitates different implementations based on the anticipated data volume and the system architecture. Domain scientists using large-scale computing facilities are therefore faced with a dilemma: they need to either perpetually maintain and tune their data processing codes to the evolving system infrastructure, or limit their investigation to analyses that can be completed in a reasonable time as datasets continue to grow in size. Declarative data processing techniques can alleviate scientists from the burden of managing how scientific data are accessed or stored. Although many declarative data management systems are actively used by scientists, these systems require time-consuming data transformations, such as loading, chunking and repartitioning, before answering any scientific query. In addition, many data management systems assume complete control of the underlying hardware, and are oblivious to optimizations and scaling opportunities that are offered through the batch execution paradigm of large-scale computing facilities. To address this gap in research, we will investigate techniques in the intersection of data management and scientific computing on how to allocate resources, and how to proactively manage parallel I/O and distributed memory. To impact scientific practice, we will develop a prototype runtime system that will augment an established scientific file format library with declarative querying capabilities for data analysis in leadership computing facilities.For further information see the project web site at: http://go.osu.edu/insitu_analysis