CAREER: Programming Environments and Runtime for Data Enabled Science

这项研究是在科学和商业的数据洪流和两个主要的计算推动力——云和百亿亿级科学系统的连接，它们与一个可互操作的运行时系统相统一。该项目有可能将这种方法转化为各种应用，从用于科学的基因组和蛋白质组数据的数据挖掘到用于商业的数据分析。研究的核心计算机科学领域——即迭代地图集体运行时、容错、数据计算协同定位和高级语言——将得到推进。此外，新的应用程序和新的软件范例将反馈到云和百亿亿级系统的架构中，可能会为国家基础设施提供特定的存储和通信选择以及新的方向。研究者将把这一新颖的研究纳入印第安纳大学的课程和研究生和本科生的研究经验，并与国内和国际合作者合作。这项工作将科学研究（计算机科学和应用）与主流商业实践（云）相结合。因此，围绕这项研究建立的课程将激励和激励学生进入劳动力市场，因此它有可能支持所需的经济发展。该研究是基于对迭代MapReduce的初步研究，并成功开发了Twister（在HPC上）和Twister4Azure（在云上）原型。该项目将构建和原型化用于数据支持科学和工程的发现环境，并开发以下组件：(1)使用Map-Collective模型作为云与集群之间可互操作的数据分析（挖掘）运行时的下一代迭代MapReduce；(2)支持并行线性代数和MapReduce等其他数据分析操作所需的多态集合操作；（三）采用发布-订阅方式的软件消息路由规模达到几万节点以上的；(4)建立在当前对象存储、数据并行文件系统（如Hadoop）和诸如Lustre的广域模型之上的存储模型，但尊重计算数据的协同定位；(5)容错模型实现为具有可配置设置的集体操作，支持迭代之间的检查点，以实现鲁棒性和单个节点故障，而不影响性能。后来的研究目标包括安全性和高级编程模型，该模型可编译为迭代MapReduce运行时。

This research is at the nexus of the data deluge in science and business and two major computing thrusts - clouds and exascale scientific systems which are unified with an interoperable runtime system. The project has the potential to transform the approach to applications that varies from data mining of genomic and proteomic data for science to data analytics for business. Computer science areas at the heart of the research - namely Iterative Map Collective runtime, fault tolerance, data-computing co-location and high level languages - will be advanced. Furthermore, the new applications enabled and new software paradigms will feed back into the architecture of cloud and exascale systems possibly suggesting particular storage and communication choices and new directions for the national infrastructure. The investigator will incorporate this novel research into courses and graduate and undergraduate research experiences at both Indiana University and with national and international collaborators. The work blends scientific research (computer science and applications) with mainstream commercial practice (clouds). Thus, curricula built around this research will motivate and inspire the entry of students into the workforce and so it has potential for supporting needed economic development.The research is based on initial research on Iterative MapReduce with successful prototypes Twister (on HPC) and Twister4Azure (on clouds). The project will architect and prototype a Discovery Environments for Data-Enabled Science and Engineering with the following components developed: (1) a next generation Iterative MapReduce using a Map-Collective model as the runtime for data analysis (mining) interoperably between clouds and clusters; (2) polymorphic collective operations needed to support parallel linear algebra and other data analysis operations such as those in MapReduce; (3) a software message routing using publish-subscribe to scale to tens of thousands of nodes or above; (4) a storage model that builds on current object stores, data parallel file systems (as in Hadoop), and wide area models like Lustre but respects compute-data co-location; (5) a fault tolerance model implemented as a Collective operation with configurable settings that supports checkpointing between iterations for robustness and individual node failure without compromising performance. Later research objectives include security and a higher-level programming model that compiles to an iterative MapReduce runtime.