Smart: a MapReduce-like framework for in-situ scientific analytics
Smart: a MapReduce-like framework for in-situ scientific analytics
复制标题
DOI：
10.1145/2807591.2807650
复制
发表时间：
2015-11
期刊：
影响因子：
--
通讯作者：
Yi Wang;G. Agrawal;Tekin Bicer;Wei Jiang
中科院分区：
文献类型：
--
作者：
Yi Wang;G. Agrawal;Tekin Bicer;Wei Jiang
现场分析最近被证明是降低科学分析的I/O和存储成本的有效方法。然而，开发有效的现场实施涉及许多挑战，包括并行化、数据移动或共享以及资源分配。基于MapReduce可以是一个适当的API用于指定科学分析应用程序的前提下，我们提出了一种新的MapReduce类似的框架，支持高效的原位科学分析，并解决了在应用MapReduce的想法在原位处理中出现的几个挑战。具体来说，我们的实现可以直接从分布式内存中加载模拟数据，并且它使用了一个经过修改的API，有助于满足现场分析的严格内存限制。该框架的设计，使分析可以从模拟程序的并行代码区域启动。我们开发了时间共享和空间共享模式，以最大限度地提高不同场景下的性能，前者甚至避免了从模拟到分析程序的任何数据复制。我们展示了我们的系统的功能，效率和可扩展性，通过使用不同的模拟和分析程序，执行多核和众核节点的集群。
In-situ analytics has lately been shown to be an effective approach to reduce both I/O and storage costs for scientific analytics. Developing an efficient in-situ implementation, however, involves many challenges, including parallelization, data movement or sharing, and resource allocation. Based on the premise that MapReduce can be an appropriate API for specifying scientific analytics applications, we present a novel MapReduce-like framework that supports efficient in-situ scientific analytics, and address several challenges that arise in applying the MapReduce idea for in-situ processing. Specifically, our implementation can load simulated data directly from distributed memory, and it uses a modified API that helps meet the strict memory constraints of in-situ analytics. The framework is designed so that analytics can be launched from the parallel code region of a simulation program. We have developed both time sharing and space sharing modes for maximizing the performance in different scenarios, with the former even avoiding any copying of data from simulation to the analytics program. We demonstrate the functionality, efficiency, and scalability of our system, by using different simulation and analytics programs, executed on clusters with multi-core and many-core nodes.