MRI: International Data Mining Grid Testbed for Research in High Performance Data Transport, Data Integration, and Data Exploration -- Instrument Development Proposal

该项目开发了一种仪器，旨在测试新的网络协议和数据服务，用于长途、高性能网络，名为Teraflow Tested(TFT)，使用先进的10 Gbs光网络并依赖于第2层光交换和第3层路由器，集成了四个地点(阿姆斯特丹、日内瓦、芝加哥-星光和芝加哥-UIC)的分布式工作站集群。该项目旨在支持关键网络技术的开发，这些技术对高性能、数据密集型计算的下一步非常重要。研究范围从低级网络协议到提供高级数据服务。TFT将由分布在两大洲的节点组成，这些节点可以传输、处理和挖掘非常大容量的数据流(Teraflow)。试验台将能够开发新的网络协议，并为Teraflow提供创新的数据集成和数据挖掘服务。这项工作涉及到设计一类新的应用程序，这些应用程序不仅可以移动查询和计算，还可以在需要时移动数据。随后，将对传统路由网络和lambda网格的协议和服务进行测试。在Lambda-Grids(通过专用密集波分复用(DWDM)光路丰富地互连的大量计算和存储资源的集合)的一般领域中，将进行以下三个具体的研究活动：Terafflow的高性能网络传输协议、Terafflow的高端到端性能和Terafflow的高性能数据服务。第一个支持开发新的网络协议，以提供更高的带宽利用率和良好的传输性能，用于具有高带宽延迟的产品光网络链路的网络。SABUL是一种基于速率的高带宽延迟可靠传输协议，基于UDP(数据传输协议)和TCP(控制流传输协议)，本文在前人工作的基础上，提出了一种UDT协议，该协议能够在保证竞争流量公平性的前提下，获得较高的吞吐量。第二个支持本地输入输出系统与极高数据速率网络协议的集成。提出了智能高速磁盘集群实验系统与TFT的集成，研究了如何将控制信息直接从并行I/O系统转发到新协议中的速率控制算法，以最大化远程并行磁盘之间的整体性能。第三个是为采矿流量开发高性能的数据服务，使用基于SOAP/XML的控制通道和单独的数据通道。该活动为terafflow可以轻松访问的数据构建了一个分布式对等存储系统，并识别数据挖掘原语以过滤和处理teraflow。更广泛的影响：TFT预计将影响国土防务、业务连续性和灾难恢复技术。博士后、研究生和本科生将参与这项研究。将在技术会议上提供有关高性能数据传输的教程。

This project, developing an instrument designed to test new network protocols and data services for long haul, high performance network called the Teraflow Testbed (TFT), integrates distributed clusters of workstation at four locations (Amsterdam, Geneva, Chicago-StarLight, and Chicago-UIC) using advanced 10 Gbs photonic networks and relying on both layer 2 optical switching and layer 3 routers. The project aims at supporting development of key network technologies important for the next step in high performance, data intensive computing. Research ranges from both low-level network protocol to offering high level data services. TFT will consist of distributed nodes over two continents that can transmit, process, and mine very high volume data flows (teraflows). The testbed will enable the development of new network protocols and innovative data integration and data mining services for teraflows. The work involves the design of a new class of applications that move not only the queries and computation, but the data when required. Subsequently, testing of the protocols and services for traditional routed networks as well as lambda grids will take place. The following three specific research activities in the general area of lambda-grids (posits collections of plentiful computing and storage resources richly interconnected by dedicated dense wavelengths division multiplexing (DWDM) optical paths) will be conducted: High Performance Network Transport Protocols for Teraflows, High End-to-End Performance for Teraflows, and High Performance Data Services for Teraflows.The first supports the development of new network protocols to provide higher bandwidth utilization and good transport performance for networks with high bandwidth-delay product optical network links. Based on previous work on SABUL, a rate based reliable transport protocol with high bandwidth delay, based on UDP (for data) and TCP (for control flow), a UDT protocol is proposed to achieve high effective throughput and still provide fairness for competing teraflows. The second supports the integration of local input-output systems with the very high data rate network protocols. Proposing the integration of an experimental system of intelligent high-speed disks connected to a cluster with the TFT, methods for relaying control information directly from the parallel I/O system to the rate control algorithm in the new protocol will be investigated to maximize overall performance between remote parallel disks. The third develops high performance data services for mining teraflows that use a SOAP/XML based control channel and a separate data channel. This activity builds a distributed peer-to-peer storage system for the data which teraflows can easily access and identifies data mining primitives to filter and process the teraflow. Broader Impacts: TFT is expected to impact homeland defense, business continuity, and disaster recovery technologies. Post docs, graduate and undergraduate students will partake in the research. Tutorials on high performance data transport will be given at technical conferences.