Studies on Remote Parallel Pipe Mechanism for Wide-Area Distributed Metacomputing

在广域网上虚拟集成多个计算资源的广域分布式元计算中，传统的基于数据并行方法的面向局域网的并行处理方法无法获得足够的性能。因此，本课题研究了远程并行管道机制，该机制可以像管道一样在远程站点上实现一个并行程序到另一个并行程序的高速并行数据传输，以减轻通信延迟带来的负面影响。由于这种机制没有包含在标准MPI函数中，因此它是该项目的原始贡献。该原型还采用了网络条带化技术，充分利用了并行计算机之间的通信带宽，并通过实验验证了其有效性。2004年，该原型被升级为可在通用PC集群上使用的集成函数库。在一个典型的PC集群中，外部通信链路仅限于一个节点，以节省分配的全局IP地址的数量，并行管道方法面临着数据分散/收集和网络条带化控制的困难。完成后的库可以将这些复杂的工作隐藏起来，不让程序员看到。其他研究成果涉及IPv6技术，利用其他节点没有外部链接，和认证机制，以实现安全的广域分布式元计算。

In wide-area distributed metacomputing by virtually integrated multiple computing resources on WAN, sufficient performance cannot be achieved through conventional LAN-oriented parallel processing methods based on the data-parallel approach. Therefore, this project focused on remote parallel pipe mechanism which performs high-speed parallel data transfer from one parallel program to another parallel program on a remote site like a pipeline, to alleviate the negative effect of the communication latency.In 2003,a prototype system successfully achieved parallel data transfer between multiple MPI programs running on distinct parallel machines. Since this mechanism is not incorporated in the standard MPI functions, it is an original contribution of this project. Network striping technique was also adopted in the prototype to fully exploit the communication band width between parallel computers, and its effectiveness was verified through experiments.In 2004,this prototype was upgraded to an integrated function library which can be used on generic PC clusters. In a typical PC cluster, external communication link is limited to only one node to save the number of global IP addresses assigned, and the parallel pipe approach faces a difficulty in data scattering/gathering and network striping control. The completed library can hide this complicated work from the programmer's view. Other research results are related to IPv6 technology to utilize other nodes having no external link, and authentication mechanism to achieve safe wide-area distributed metacomputing.