An RDMA-based memory management architecture for large-scale distributed computation frameworks

一种基于RDMA的大规模分布式计算框架内存管理架构

• 批准号: 474648-2014
• 负责人: Wong, Bernard
• 金额: $ 1.55万
• 依托单位: University of Waterloo
• 依托单位国家: 加拿大
• 项目类别: Engage Grants Program
• 财政年份: 2014
• 资助国家: 加拿大
• 起止时间: 2014-01-01 至 2015-12-31
• 项目状态: 已结题
• 来源: https://www.nserc-crsng.gc.ca/ase-oro/Details-Detailles_eng.asp?id=565025
• 关键词: RDMA; based; memory; management; architecture

Extracting knowledge from Big Data sources can be the key to the next big technological breakthrough. Current Big Data processing approaches rely on high-performance, distributed computation frameworks that leverage the combined resources of a large computing cluster to reduce the required processing time. Unfortunately, current frameworks manage resources from individual machines separately and are therefore unable to take full advantage of a cluster's resources. In this research proposal, we investigate a new approach to manage resources in distributed computation frameworks that takes advantage of high-speed datacenter networks to allow resources across machines to be managed as a single resource pool. By more effectively using resources, our approach can avoid recomputing intermediate results and minimize the use of slow persistent storage systems. This can lead to significant performance improvements that can enable processing of new Big Data sources that are beyond our current capabilities. In addition to changes to the memory management system, this work will also investigate the use of more efficient network communication methods that can reduce the data communication overhead between servers in a computation cluster. Overall, this work can help Canadian IT companies by improving their ability to process Big Data sources, which can give them a significant competitive advantage over their competitors.

从大数据源中提取知识可能是下一个重大技术突破的关键。当前的大数据处理方法依赖于高性能的分布式计算框架，该框架利用大型计算集群的组合资源来减少所需的处理时间。不幸的是，当前的框架单独管理各个机器的资源，因此无法充分利用集群的资源。在本研究提案中，我们研究了一种在分布式计算框架中管理资源的新方法，该方法利用高速数据中心网络来允许跨机器的资源作为单个资源池进行管理。通过更有效地利用资源，我们的方法可以避免重新计算中间结果，并最大限度地减少慢速持久存储系统的使用。这可以带来显着的性能改进，从而能够处理超出我们当前能力的新大数据源。除了对内存管理系统的更改之外，这项工作还将研究使用更有效的网络通信方法，这些方法可以减少计算集群中服务器之间的数据通信开销。总体而言，这项工作可以通过提高加拿大 IT 公司处理大数据源的能力来帮助他们，这可以使他们比竞争对手获得显着的竞争优势。