CRII: III: Declarative array processing for large-scale scientific analyses

科学家们通过运行在数十万个处理核心上的数据密集型分析来理解复杂的自然现象。然而，并行快速探索非常大的数据集以获得洞察力是具有挑战性的。分析大于内存的数据集暴露了存储层次结构的复杂性，需要根据预期的数据量和系统架构实现不同的实现。因此，使用大规模计算设施的领域科学家面临着两难境地：他们要么需要永远维护和调整他们的数据处理代码以适应不断发展的系统基础设施，要么将他们的调查限制在可以在合理的时间内完成的分析，因为数据集的规模继续增长。声明性数据处理技术可以减轻科学家管理如何访问或存储科学数据的负担。尽管科学家们积极使用许多声明性数据管理系统，但这些系统在回答任何科学查询之前都需要进行耗时的数据转换，例如加载、分块和重新分区。此外，许多数据管理系统假定完全控制底层硬件，而忽略了通过大规模计算设施的批处理执行范例提供的优化和扩展机会。为了解决研究中的这一差距，我们将研究数据管理和科学计算的交叉技术，研究如何分配资源，以及如何主动管理并行I/O和分布式内存。为了影响科学实践，我们将开发一个原型运行时系统，该系统将增强已建立的科学文件格式库，并具有用于领导计算设施的数据分析的声明性查询功能。欲了解更多信息，请参阅该项目的网站：http://go.osu.edu/insitu_analysis

Scientists understand complex natural phenomena through data-intensive analyses that run on hundreds of thousands of processing cores. Quickly exploring very large datasets in parallel for insights, however, is challenging. Analyzing larger-than-memory datasets exposes the intricacies of the storage hierarchy and necessitates different implementations based on the anticipated data volume and the system architecture. Domain scientists using large-scale computing facilities are therefore faced with a dilemma: they need to either perpetually maintain and tune their data processing codes to the evolving system infrastructure, or limit their investigation to analyses that can be completed in a reasonable time as datasets continue to grow in size. Declarative data processing techniques can alleviate scientists from the burden of managing how scientific data are accessed or stored. Although many declarative data management systems are actively used by scientists, these systems require time-consuming data transformations, such as loading, chunking and repartitioning, before answering any scientific query. In addition, many data management systems assume complete control of the underlying hardware, and are oblivious to optimizations and scaling opportunities that are offered through the batch execution paradigm of large-scale computing facilities. To address this gap in research, we will investigate techniques in the intersection of data management and scientific computing on how to allocate resources, and how to proactively manage parallel I/O and distributed memory. To impact scientific practice, we will develop a prototype runtime system that will augment an established scientific file format library with declarative querying capabilities for data analysis in leadership computing facilities.For further information see the project web site at: http://go.osu.edu/insitu_analysis