An RDMA-based memory management architecture for large-scale distributed computation frameworks

点击翻译按钮获取中文摘要

Extracting knowledge from Big Data sources can be the key to the next big technological breakthrough. Current Big Data processing approaches rely on high-performance, distributed computation frameworks that leverage the combined resources of a large computing cluster to reduce the required processing time. Unfortunately, current frameworks manage resources from individual machines separately and are therefore unable to take full advantage of a cluster's resources. In this research proposal, we investigate a new approach to manage resources in distributed computation frameworks that takes advantage of high-speed datacenter networks to allow resources across machines to be managed as a single resource pool. By more effectively using resources, our approach can avoid recomputing intermediate results and minimize the use of slow persistent storage systems. This can lead to significant performance improvements that can enable processing of new Big Data sources that are beyond our current capabilities. In addition to changes to the memory management system, this work will also investigate the use of more efficient network communication methods that can reduce the data communication overhead between servers in a computation cluster. Overall, this work can help Canadian IT companies by improving their ability to process Big Data sources, which can give them a significant competitive advantage over their competitors.