II-NEW: A Virtualized Cluster Testbed to Support Research In Large Memory And Data Intensive Applications

这项研究的目标是开发和部署一个托管多TB集体内存的集群试验台。其方法是开发一个分布式内核级内存虚拟化层--一个轻量级分布式系统，将网络上多台机器的集体内存汇集在一起。试验床的智力优势在于使涉及高性能和数据密集型计算的未经修改的大内存应用程序的研究活动成为可能，这些应用程序的内存需求通常无法在任何一台商用机器上满足。试验台利用了一个关键的观察结果，即通过低延迟千兆以太网访问另一台机器的内存比访问本地磁盘快一个数量级。在测试床上执行的应用程序将不需要任何代码修改、重新编译或重新链接。试验床将具有自动化和可扩展的内存管理、分布式资源发现、巨型帧支持和容错功能。更广泛的影响将是满足研究人员对负担得起的多TB内存试验台的共同迫切需求，该试验台透明地支持任何未经修改的应用程序。一旦成熟，试验床将向促进进一步合作的其他研究人员开放。该项目将通过让一些少数族裔、女性和本科生参与试验床的获取和开发来促进多样性。试验台还将对宾厄姆顿大学的高性能和分布式系统教育计划产生重大影响。PI将利用课程项目的试验台平台提供3D成像和分析、数据挖掘、安全和虚拟化方面的课程。

The objective of this research is to develop and deploy a cluster testbed hosting multiple terabytes of collective memory. The approach is to develop a distributed kernel-level memory virtualization layer -- a lightweight distributed system that pools together the collective memory of multiple machines across the network.The intellectual merit of the testbed is in enabling research activities involving unmodified large memory applications in high performance and data intensive computing whose memory requirements cannot be normally accommodated within any single commodity machine. The testbed exploits the key observation that it is an order of magnitude faster to access the memory of another machine over low-latency gigabit Ethernet than it is to access the local disk. Applications executing over the testbed will not require any code modification, recompilation, or relinking. The testbed will feature automated and scalable memory management, distributed resource discovery, jumbo frame support, and fault-tolerance. The broader impact will be in fulfilling a common compelling need among researchers for an affordable multi-terabyte memory testbed that transparently supports any unmodified application. Once mature, the testbed will be made accessible to other researchers fostering further collaboration. The project will promote diversity by involving a number of minority, women, and undergraduate students in testbed acquisition and development. The testbed will also have a major impact on the high-performance and distributed systems education program at Binghamton University. The PIs will offer courses in 3D imaging and analysis, data mining, security, and virtualization utilizing the testbed platform for course projects.