An RDMA-based memory management architecture for large-scale distributed computation frameworks

从大数据源中提取知识可能是下一个重大技术突破的关键。当前的大数据处理方法依赖于高性能的分布式计算框架，该框架利用大型计算集群的组合资源来减少所需的处理时间。不幸的是，当前的框架分别管理来自各个机器的资源，因此无法充分利用集群的资源。在这项研究建议中，我们研究了一种新的方法来管理分布式计算框架中的资源，该框架利用高速数据中心网络，允许跨机器的资源作为一个单一的资源池进行管理。通过更有效地利用资源，我们的方法可以避免重新计算中间结果，并尽量减少使用缓慢的持久存储系统。这可以带来显著的性能改进，从而能够处理超出我们当前能力的新大数据源。除了内存管理系统的变化，这项工作还将研究使用更有效的网络通信方法，可以减少计算集群中的服务器之间的数据通信开销。总的来说，这项工作可以帮助加拿大IT公司提高处理大数据源的能力，这可以使他们在竞争对手中获得显着的竞争优势。

Extracting knowledge from Big Data sources can be the key to the next big technological breakthrough. Current Big Data processing approaches rely on high-performance, distributed computation frameworks that leverage the combined resources of a large computing cluster to reduce the required processing time. Unfortunately, current frameworks manage resources from individual machines separately and are therefore unable to take full advantage of a cluster's resources. In this research proposal, we investigate a new approach to manage resources in distributed computation frameworks that takes advantage of high-speed datacenter networks to allow resources across machines to be managed as a single resource pool. By more effectively using resources, our approach can avoid recomputing intermediate results and minimize the use of slow persistent storage systems. This can lead to significant performance improvements that can enable processing of new Big Data sources that are beyond our current capabilities. In addition to changes to the memory management system, this work will also investigate the use of more efficient network communication methods that can reduce the data communication overhead between servers in a computation cluster. Overall, this work can help Canadian IT companies by improving their ability to process Big Data sources, which can give them a significant competitive advantage over their competitors.