MRI: International Data Mining Grid Testbed for Research in High Performance Data Transport, Data Integration, and Data Exploration -- Instrument Development Proposal

该项目开发了一种仪器，用于测试长途高性能网络的新网络协议和数据服务，称为terflow测试台（TFT），使用先进的10gb光子网络，并依靠第二层光交换和第三层路由器，集成了四个地点（阿姆斯特丹，日内瓦，芝加哥- starlight和芝加哥- uic）的分布式工作站集群。该项目旨在支持关键网络技术的发展，这些技术对高性能、数据密集型计算的下一步发展至关重要。研究范围从低级网络协议到提供高级数据服务。TFT将由分布在两大洲的分布式节点组成，这些节点可以传输、处理和挖掘非常大容量的数据流（teraflows）。该测试平台将为太流开发新的网络协议、创新的数据集成和数据挖掘服务。这项工作涉及设计一类新的应用程序，这些应用程序不仅可以移动查询和计算，还可以在需要时移动数据。随后，将对传统路由网络以及lambda网格的协议和服务进行测试。在lambda-grids（通过专用密集波分复用（DWDM）光路互连的大量计算和存储资源的集合）的一般领域，将进行以下三个具体的研究活动：用于Teraflows的高性能网络传输协议，用于Teraflows的高端到端性能和用于Teraflows的高性能数据服务。第一种支持开发新的网络协议，为具有高带宽延迟产品光网络链路的网络提供更高的带宽利用率和良好的传输性能。SABUL是一种基于速率的高带宽延迟可靠传输协议，基于UDP（用于数据）和TCP（用于控制流），在此基础上提出了一种UDT协议，以实现高效吞吐量，并为竞争的太流提供公平性。第二种支持本地输入输出系统与非常高的数据速率网络协议的集成。提出将智能高速磁盘集群连接实验系统与TFT集成，研究将控制信息从并行I/O系统直接传递到新协议中的速率控制算法的方法，以最大限度地提高远程并行磁盘之间的整体性能。第三部分开发用于挖掘使用基于SOAP/XML的控制通道和独立数据通道的太流的高性能数据服务。该活动为数据构建了一个分布式的点对点存储系统，太流可以很容易地访问和识别数据挖掘原语，以过滤和处理太流。更广泛的影响：预计TFT将影响国土防御、业务连续性和灾难恢复技术。博士后、研究生和本科生将参与研究。有关高性能数据传输的教程将在技术会议上提供。

This project, developing an instrument designed to test new network protocols and data services for long haul, high performance network called the Teraflow Testbed (TFT), integrates distributed clusters of workstation at four locations (Amsterdam, Geneva, Chicago-StarLight, and Chicago-UIC) using advanced 10 Gbs photonic networks and relying on both layer 2 optical switching and layer 3 routers. The project aims at supporting development of key network technologies important for the next step in high performance, data intensive computing. Research ranges from both low-level network protocol to offering high level data services. TFT will consist of distributed nodes over two continents that can transmit, process, and mine very high volume data flows (teraflows). The testbed will enable the development of new network protocols and innovative data integration and data mining services for teraflows. The work involves the design of a new class of applications that move not only the queries and computation, but the data when required. Subsequently, testing of the protocols and services for traditional routed networks as well as lambda grids will take place. The following three specific research activities in the general area of lambda-grids (posits collections of plentiful computing and storage resources richly interconnected by dedicated dense wavelengths division multiplexing (DWDM) optical paths) will be conducted: High Performance Network Transport Protocols for Teraflows, High End-to-End Performance for Teraflows, and High Performance Data Services for Teraflows.The first supports the development of new network protocols to provide higher bandwidth utilization and good transport performance for networks with high bandwidth-delay product optical network links. Based on previous work on SABUL, a rate based reliable transport protocol with high bandwidth delay, based on UDP (for data) and TCP (for control flow), a UDT protocol is proposed to achieve high effective throughput and still provide fairness for competing teraflows. The second supports the integration of local input-output systems with the very high data rate network protocols. Proposing the integration of an experimental system of intelligent high-speed disks connected to a cluster with the TFT, methods for relaying control information directly from the parallel I/O system to the rate control algorithm in the new protocol will be investigated to maximize overall performance between remote parallel disks. The third develops high performance data services for mining teraflows that use a SOAP/XML based control channel and a separate data channel. This activity builds a distributed peer-to-peer storage system for the data which teraflows can easily access and identifies data mining primitives to filter and process the teraflow. Broader Impacts: TFT is expected to impact homeland defense, business continuity, and disaster recovery technologies. Post docs, graduate and undergraduate students will partake in the research. Tutorials on high performance data transport will be given at technical conferences.