Parallel and Distributed Computing for Real-time Analytics

用于实时分析的并行和分布式计算

• 批准号: RGPIN-2015-04934
• 负责人: Wagner, Alan
• 金额: $ 1.31万
• 依托单位: University of British Columbia
• 依托单位国家: 加拿大
• 项目类别: Discovery Grants Program - Individual
• 财政年份: 2018
• 资助国家: 加拿大
• 起止时间: 2018-01-01 至 2019-12-31
• 项目状态: 已结题
• 来源: https://www.nserc-crsng.gc.ca/ase-oro/Details-Detailles_eng.asp?id=667931
• 关键词: Parallel; Distributed; Computing; Real; time

The goal of my research is to create a runtime system that can scale-out to large size clusters of multicore machines for in-memory computation of analytics on big data. Many of the existing distributed systems do not take advantage of the tools and techniques developed in high performance computing for addressing the performance issues that ultimately limits scalability. MPI (Message-Passing Interface) is a message-passing library that is the de-facto standard in parallel programming for high performance computing. I intend to develop new MPI middleware along with associated tools as the basis for a scalable runtime system for computing analytics.***Scaling-out requires the ability to express massive amounts of concurrency and ultimately relies on message-passing. As such, the starting point for this work will be our FG-MPI (Fine-Grain MPI (Message-Passing Interface)) system, which we have shown can scale to thousands and millions of processes. FG-MPI extends MPICH, an MPI runtime system that is used to derive most of the commercial versions of MPI. This work will require extending the existing FG-MPI middleware as well as introducing new tools and techniques for programming at large scale.***The research involves (a) investigating further improvements to the MPI middleware to support multicore machines, (b) developing tools to support an "actor-like" programming environment for specifying the composition of processes and the development of service-oriented data structures for MPI, (c) following a million process methodology to develop techniques for developing and deploying programs at scale, (d) developing innovative message-based data structures and algorithms to support knowledge-based analysis and reasoning on big data. ***This research directly addresses the two main trends in computer technology: the appearance of multicore and cluster/cloud computing. In both cases --- the future is parallel! The central problem is how to use multicore to scale-up or how to use more machines to scale-out to provide the performance needed to process and compute analytics on the vast amount of data that is available today.********

我的研究目标是创建一个运行时系统，该系统可以扩展到大型多核机器集群，用于大数据分析的内存计算。许多现有的分布式系统没有利用高性能计算中开发的工具和技术来解决最终限制可伸缩性的性能问题。MPI（消息传递接口）是一个消息传递库，它是用于高性能计算的并行编程中的事实上的标准。我打算开发新的MPI中间件以及相关的工具，作为计算分析的可伸缩运行时系统的基础。横向扩展需要表达大量并发性的能力，并最终依赖于消息传递。因此，这项工作的起点将是我们的FG-MPI（细粒度MPI（消息传递接口））系统，我们已经展示了该系统可以扩展到数千甚至数百万个进程。FG-MPI扩展了MPICH， MPICH是一个MPI运行时系统，用于派生大多数MPI的商业版本。这项工作将需要扩展现有的FG-MPI中间件，并为大规模编程引入新的工具和技术。该研究包括(a)调查进一步改进MPI中间件以支持多核机器，(b)开发工具以支持“类似参与者”的编程环境，用于指定MPI的过程组合和面向服务的数据结构的开发，(c)遵循百万过程方法来开发用于大规模开发和部署程序的技术。(d)开发创新的基于消息的数据结构和算法，以支持基于知识的大数据分析和推理。本研究直接涉及计算机技术的两个主要趋势：多核和集群/云计算的出现。在这两种情况下——未来是平行的！核心问题是如何使用多核来扩展，或者如何使用更多的机器来扩展，以提供处理和计算分析所需的性能，对今天可用的大量数据。********