EAGER: NDN-Hadoop: Exploring Applicability of NDN for Big-Data Computing

海量数据的大规模分布式处理是大数据时代的基础技术。Apache Hadoop是一个开源软件平台，已成为运行多种类型分布式应用程序的首选技术，如Web索引、数据挖掘、商业智能分析、机器学习、科学模拟和生物信息学研究。这种广泛的应用要求Hadoop在为每个应用提供高性能的同时，必须灵活地适应不同的网络环境，这导致了复杂的协议设计、实现、大量的参数需要调优，而且在真实的部署中性能往往不尽如人意。命名数据网络（NDN）是一种新兴的网络架构，它将重点从点对点通信转移到一般的内容检索，比传统的TCP/IP（互联网的通信协议）更适合大数据计算。NDN可以在简化部署、提高Hadoop系统的健壮性和性能方面提供巨大的好处。该项目研究了将NDN和Hadoop集成以构建NDN-Hadoop系统的可行性，该系统将为大数据计算提供强大，高效和可扩展的基础。由于Hadoop是一个复杂的生态系统，而NDN正在积极开发中，关于它们如何协同工作以及如何最大限度地利用潜在的好处，还有很多问题需要回答。更具体地说，该项目将（1）通过了解NDN的好处和量化运行在NDN上的Hadoop的性能增益来评估NDN在Hadoop基础设施中的适用性，（2）设计新颖的Hadoop机制，以充分利用NDN及其提供的信息，以获得更好的性能和可靠性，（3）在NDN上开发和测试一个完整的Hadoop系统，为未来的研究提供平台。将NDN引入到企业级网络和计算中将对大型数据中心的设计产生深远的影响，分布式计算基础设施和多个研究社区。该项目将产生第一个以数据为中心的Hadoop系统，这将有利于使用Hadoop的各种应用程序。研究结果还可以转化为其他大数据计算平台，进一步扩大影响。它不仅为网络和系统社区提供了新的研究方向，而且还为科学计算社区和数据分析提供了性能增强。本研究所取得的经验将反馈到NDN的研究中，为未来Internet的发展做出贡献。该项目为研究生和本科生提供了很好的教育和培训机会。

Large-scale distributed processing of huge amount of data is the underpinning technology in the era of Big Data. The Apache Hadoop, an open-source software platform, has become the go-to technology for running multiple types of distributed applications such as Web indexing, data mining, business intelligence analysis, machine learning, scientific simulation, and bioinformatics research. This wide range of applications requires Hadoop to be flexible and adaptive to different network environments while providing high performance for each application, resulting in complicated protocol design, implementation, a large number of parameters to tune, and often unsatisfactory performance in real deployment. An emerging network architecture, Named Data Networking (NDN) shifts the focus from point-to-point communication to general content retrieval, a much better fit to Big Data computing than traditional TCP/IP, the communication protocol of the internet. NDN can potentially offer tremendous benefits in simplifying deployments, improving robustness and performance for Hadoop systems. This project investigates the feasibility of integrating NDN and Hadoop to build an NDN-Hadoop system that will provide a robust, efficient, and scalable foundation for Big Data computing. Since Hadoop is a complicated ecosystem and NDN is being actively developed, there are many questions to be answered regarding how they may work together and how to exploit the potential benefits to the greatest extent. More specifically, this project will (1) evaluate the applicability of NDN in Hadoop infrastructure by understanding its benefits and quantifying performance gains of Hadoop running on top of NDN, (2) design novel Hadoop mechanisms to take full advantage of NDN and the information it provides for better performance and reliability, and (3) develop and test a complete system of Hadoop running on top of NDN to create a platform for future research.Introducing NDN into datacenter-scale networking and computing will have profound impacts on the design of large data centers, distributed computing infrastructures, and multiple research communities. The project will produce the first data-centric Hadoop system, which will benefit all kinds of applications that use Hadoop. The results can also be translated into other Big Data computing platforms, broadening the impacts even further. It provides not only new research directions to the network and system community, but also performance enhancements to scientific computing community and data analytics. The experiences gained in this project will feedback to NDN research, contributing to the development of future Internet. The project offers great education and training opportunities for graduate and undergraduate students.