Studies on Remote Parallel Pipe Mechanism for Wide-Area Distributed Metacomputing

在广域网上虚拟集成多个计算资源的广域分布式元计算中，传统的基于数据并行方法的面向局域网的并行处理方法无法获得足够的性能。因此，本项目重点研究远程并行管道机制，以管道的形式在远程站点上实现从一个并行程序到另一个并行程序的高速并行数据传输，以减轻通信延迟带来的负面影响。2003年，一个原型系统成功地实现了运行在不同并行机上的多个MPI程序之间的并行数据传输。由于该机制没有包含在标准MPI功能中，因此它是本项目的原始贡献。为了充分利用并行计算机之间的通信带宽，原型机还采用了网络条带化技术，并通过实验验证了其有效性。2004年，该原型升级为可在通用PC集群上使用的集成函数库。在典型的PC集群中，为了节省分配的全局IP地址数量，外部通信链路被限制在一个节点上，并行管道方式面临着数据分散/收集和网络分条控制的困难。完成的库可以在程序员的视图中隐藏这些复杂的工作。其他研究成果涉及IPv6技术，利用无外部链接的其他节点，以及认证机制实现安全的广域分布式元计算。

In wide-area distributed metacomputing by virtually integrated multiple computing resources on WAN, sufficient performance cannot be achieved through conventional LAN-oriented parallel processing methods based on the data-parallel approach. Therefore, this project focused on remote parallel pipe mechanism which performs high-speed parallel data transfer from one parallel program to another parallel program on a remote site like a pipeline, to alleviate the negative effect of the communication latency.In 2003,a prototype system successfully achieved parallel data transfer between multiple MPI programs running on distinct parallel machines. Since this mechanism is not incorporated in the standard MPI functions, it is an original contribution of this project. Network striping technique was also adopted in the prototype to fully exploit the communication band width between parallel computers, and its effectiveness was verified through experiments.In 2004,this prototype was upgraded to an integrated function library which can be used on generic PC clusters. In a typical PC cluster, external communication link is limited to only one node to save the number of global IP addresses assigned, and the parallel pipe approach faces a difficulty in data scattering/gathering and network striping control. The completed library can hide this complicated work from the programmer's view. Other research results are related to IPv6 technology to utilize other nodes having no external link, and authentication mechanism to achieve safe wide-area distributed metacomputing.