Research on Peer-to-peer large-scale data processing on the Grid

点击翻译按钮获取中文摘要

The aim of this project is to develop basic technologies for the petabyte scale data processing. The goal is to bear the actual data processing from the LHC/ATLAS detector, as a part of the 'Gfarm' data grid middleware project. We have studied the following themes.1). Wide area peer-to-peer federation and data transfer among the large clusters on the Grid2). Programming, performance analyses and simulation techniques on the peer-to-peer data Grid.3). High performance large scale data management for the data Grid.1) We have promoted research on wide area large scale data transfer, and participated in the Bandwidth Challenge 2002, 2003 and StorCloud 2005 held in the United States to demonstrate the performance of large data transfer. Through the challenges, we have succeeded in running real applications on the GFarm and got the outlook for the realization of the super large scale data centers and the super scale data analyses for international collaborative experiments.2) We have developed a Grid simulator named 'Bricks' to construct the performance models and to perform analyses of the data Grid. With that, we made lots of simulations of the data Grid and investigated characteristics of the data Grid and the GFarm architecture. We have also developed a Grid programming environment 'Jay', which is portable and suitable for the peer-to-peer Grid environments3) We have developed a widely distributed file system that can avoid access concentration on a small number of nodes on the data Grid. We have implemented the system as an extension to GFarm, and inspected the validity of the system.