CC-NIE Integration: Bridging, Transferring and Analyzing Big Data over 10Gbps Campus-Wide Software Defined Networks

在路易斯安那州立大学进行的CC-NIE集成项目正在开发一个网络基础设施，该基础设施将六个不同的大型科学研究小组与路易斯安那州立大学的高性能计算（HPC）集群集成在一起，使用软件定义的网络技术和基于Hadoop和MPI（消息传递接口）的并行框架。该项目包括三个目标：（i）使用OpenFlow交换机和控制器构建10 Gbps软件定义网络（SDN），为每个组提供多个虚拟网络切片;（ii）通过SDN网络上的多个光路径传输大数据，实现最高20 Gbps的聚合磁盘到磁盘传输速率;及（iii）通过Hadoop和MPI技术开发数据密集型分布式计算框架，在HPC集群上并行处理大量作业，从而分析大数据。这三个组成部分与一个门户网站服务和一个启用GENI的网络管理系统相结合。为了实现20 Gbps的高磁盘到磁盘传输速率，该项目与三星电子建立了工业合作伙伴关系，提供70 TB固态硬盘（SSD）存储并优化I/O带宽。网络基础设施加速了当前大数据研究项目的发展，这些项目涵盖了广泛的研究领域，包括生物学和兽医学院的基因测序研究，计算化学，计算机科学的大数据挖掘，土木和环境工程的沿海灾害模拟研究。最后，该项目将建立一种方法来构建集成的网络基础设施，包括高速网络，高性能计算和大数据科学与工程的高速存储。

This CC-NIE Integration project performed at the Louisiana State University is developing a cyber-infrastructure integrating six different large scale scientific research groups with high performance computing (HPC) clusters at LSU using software defined network technology and Hadoop and MPI (Message Passing Interface) based parallel frameworks. The project consists of three objectives: (i) Building 10Gbps software-defined network (SDN) with OpenFlow switches and controllers to provide multiple virtual network slices to each group; (ii) Transferring Big Data with automatically tuned operation through multiple optical paths over the SDN network achieving at most 20Gbps of aggregated disk-to-disk transfer rate; and (iii) Analyzing Big Data by developing data-intensive distributed computing frameworks with Hadoop and MPI technologies parallelizing large number of jobs over HPC clusters. Those three components are integrated with a web portal service and a GENI-enabled network management system. To achieve high disk-to-disk transfer rate at 20Gbps, the project has an industrial partnership with Samsung Electronics that contributes 70TB Solid State Drive (SSD) storage and optimizes the I/O bandwidth. The cyber-infrastructure accelerates the current Big Data research projects spanning a wide range of research areas including gene sequencing research at Biology and Vet School, computational chemistry, big data mining at Computer Science, coastal hazard simulation research at civil and environmental engineering. In the end, the project will establish a methodology to build integrated cyber-infrastructures consisting of high speed networks, high performance computing, and high speed storage for the Big Data Science and Engineering.