CAREER: Programming Environments and Runtime for Data Enabled Science

这项研究是在科学和商业中的数据洪流和两个主要的计算推动力-云和亿级科学系统-与可互操作的运行时系统统一的结合点上进行的。该项目有可能将方法转变为各种应用，从用于科学的基因组和蛋白质数据的数据挖掘到用于商业的数据分析。这项研究的核心计算机科学领域--即迭代地图集合运行时、容错、数据计算协同定位和高级语言--将得到推进。此外，启用的新应用程序和新的软件范例将反馈到云和亿级系统的体系结构中，可能会为国家基础设施提供特定的存储和通信选择以及新的方向。研究人员将把这项新颖的研究纳入印第安纳大学的课程以及研究生和本科生的研究经验，并与国内和国际合作者合作。这项工作将科学研究(计算机科学和应用)与主流商业实践(云)相结合。因此，围绕这项研究建立的课程将激励和激励学生进入劳动力大军，因此它具有支持所需经济发展的潜力。这项研究基于迭代MapReduce的初步研究，成功的原型是Twister(基于HPC)和Twister4Azure(基于云)。该项目将为数据使能的科学和工程设计一个发现环境并建立原型，开发以下组件：(1)使用地图-集合模型作为云和集群之间可互操作的数据分析(挖掘)的运行时的下一代迭代MapReduce；(2)支持并行线性代数和其他数据分析操作所需的多态集合操作，例如在MapReduce中的那些操作；(3)使用发布-订阅的软件消息路由，以扩展到数万个节点或更高；(4)构建在当前对象存储、数据并行文件系统(如Hadoop中)和广域模型(如Lustre)上的存储模型；(5)作为集合操作实现的容错模型，具有可配置的设置，支持在迭代之间设置检查点以实现健壮性和单个节点故障，而不会影响性能。以后的研究目标包括安全性和高级编程模型，该模型编译为迭代的MapReduceTM运行时。

This research is at the nexus of the data deluge in science and business and two major computing thrusts - clouds and exascale scientific systems which are unified with an interoperable runtime system. The project has the potential to transform the approach to applications that varies from data mining of genomic and proteomic data for science to data analytics for business. Computer science areas at the heart of the research - namely Iterative Map Collective runtime, fault tolerance, data-computing co-location and high level languages - will be advanced. Furthermore, the new applications enabled and new software paradigms will feed back into the architecture of cloud and exascale systems possibly suggesting particular storage and communication choices and new directions for the national infrastructure. The investigator will incorporate this novel research into courses and graduate and undergraduate research experiences at both Indiana University and with national and international collaborators. The work blends scientific research (computer science and applications) with mainstream commercial practice (clouds). Thus, curricula built around this research will motivate and inspire the entry of students into the workforce and so it has potential for supporting needed economic development.The research is based on initial research on Iterative MapReduce with successful prototypes Twister (on HPC) and Twister4Azure (on clouds). The project will architect and prototype a Discovery Environments for Data-Enabled Science and Engineering with the following components developed: (1) a next generation Iterative MapReduce using a Map-Collective model as the runtime for data analysis (mining) interoperably between clouds and clusters; (2) polymorphic collective operations needed to support parallel linear algebra and other data analysis operations such as those in MapReduce; (3) a software message routing using publish-subscribe to scale to tens of thousands of nodes or above; (4) a storage model that builds on current object stores, data parallel file systems (as in Hadoop), and wide area models like Lustre but respects compute-data co-location; (5) a fault tolerance model implemented as a Collective operation with configurable settings that supports checkpointing between iterations for robustness and individual node failure without compromising performance. Later research objectives include security and a higher-level programming model that compiles to an iterative MapReduce runtime.