Research on Peer-to-peer large-scale data processing on the Grid

该项目的目的是开发PB级数据处理的基本技术。目标是承担来自LHC/ATLAS探测器的实际数据处理，作为‘Garm’数据网格中间件项目的一部分。我们研究了以下几个主题。广域对等联合和GRID2上大型集群之间的数据传输)。3)对等数据网格的编程、性能分析和仿真技术。面向数据网格的高性能大规模数据管理1)我们推动了广域大规模数据传输的研究，并参加了在美国举行的2002、2003年带宽挑战赛和StorCloud 2005，展示了大数据传输的性能。通过这些挑战，我们成功地在GFarm上运行了真实的应用程序，并对超大规模数据中心的实现和国际协作实验的超大规模数据分析进行了展望。2)开发了一个名为Brick的网格模拟器，用于构建数据网格的性能模型和进行数据网格分析。在此基础上，对数据网格进行了大量的仿真，研究了数据网格和GFarm体系结构的特点。我们还开发了一个可移植的、适用于对等网格环境的网格编程环境Jay。3)我们开发了一个广泛分布的文件系统，可以避免访问集中在数据网格上的少数节点上。我们实现了该系统作为GFARM的扩展，并对系统的有效性进行了检验。

The aim of this project is to develop basic technologies for the petabyte scale data processing. The goal is to bear the actual data processing from the LHC/ATLAS detector, as a part of the 'Gfarm' data grid middleware project. We have studied the following themes.1). Wide area peer-to-peer federation and data transfer among the large clusters on the Grid2). Programming, performance analyses and simulation techniques on the peer-to-peer data Grid.3). High performance large scale data management for the data Grid.1) We have promoted research on wide area large scale data transfer, and participated in the Bandwidth Challenge 2002, 2003 and StorCloud 2005 held in the United States to demonstrate the performance of large data transfer. Through the challenges, we have succeeded in running real applications on the GFarm and got the outlook for the realization of the super large scale data centers and the super scale data analyses for international collaborative experiments.2) We have developed a Grid simulator named 'Bricks' to construct the performance models and to perform analyses of the data Grid. With that, we made lots of simulations of the data Grid and investigated characteristics of the data Grid and the GFarm architecture. We have also developed a Grid programming environment 'Jay', which is portable and suitable for the peer-to-peer Grid environments3) We have developed a widely distributed file system that can avoid access concentration on a small number of nodes on the data Grid. We have implemented the system as an extension to GFarm, and inspected the validity of the system.