Research on Peer-to-peer large-scale data processing on the Grid

网格上点对点大规模数据处理研究

• 批准号: 13224034
• 负责人: MATSUOKA Satoshi
• 金额: $ 72.45万
• 依托单位: Tokyo Institute of Technology
• 依托单位国家: 日本
• 项目类别: Grant-in-Aid for Scientific Research on Priority Areas
• 财政年份: 2001
• 资助国家: 日本
• 起止时间: 2001 至 2005
• 项目状态: 已结题
• 来源: https://kaken.nii.ac.jp/en/grant/KAKENHI-PROJECT-13224034/
• 关键词: Data Grid; Peer-to-Peer; Data Cluster; Grid; Distributed Filesystem; Grid Simulation; 分散ファイル; Grid; ペタバイト級データ; コモディティクラスタ; Gfarm; Bandwidth Challenge; レプリカシステム; モニタリングシステム; ハイパフォーマンスコンピューティング技術; グリッド計算; PCクラスタ; 大規模データ処理; インターネット高度化技術; L

The aim of this project is to develop basic technologies for the petabyte scale data processing. The goal is to bear the actual data processing from the LHC/ATLAS detector, as a part of the 'Gfarm' data grid middleware project. We have studied the following themes.1). Wide area peer-to-peer federation and data transfer among the large clusters on the Grid2). Programming, performance analyses and simulation techniques on the peer-to-peer data Grid.3). High performance large scale data management for the data Grid.1) We have promoted research on wide area large scale data transfer, and participated in the Bandwidth Challenge 2002, 2003 and StorCloud 2005 held in the United States to demonstrate the performance of large data transfer. Through the challenges, we have succeeded in running real applications on the GFarm and got the outlook for the realization of the super large scale data centers and the super scale data analyses for international collaborative experiments.2) We have developed a Grid simulator named 'Bricks' to construct the performance models and to perform analyses of the data Grid. With that, we made lots of simulations of the data Grid and investigated characteristics of the data Grid and the GFarm architecture. We have also developed a Grid programming environment 'Jay', which is portable and suitable for the peer-to-peer Grid environments3) We have developed a widely distributed file system that can avoid access concentration on a small number of nodes on the data Grid. We have implemented the system as an extension to GFarm, and inspected the validity of the system.

该项目的目标是开发PB级数据处理的基础技术。目标是承担LHC/ATLAS探测器的实际数据处理，作为“Gfarm”数据网格中间件项目的一部分。我们研究了以下主题：（1）。广域对等联盟和网格上大型集群之间的数据传输2）。对等数据网格的编程、性能分析和仿真技术。1）推进广域大规模数据传输的研究，参加了在美国举办的2002年、2003年带宽挑战赛（Bandwidth Challenge 2002）、2003年、2005年StorCloud 2005等，展示了大规模数据传输性能。通过这些挑战，我们成功地在GFarm上运行了真实的应用程序，并对超大规模数据中心的实现和超大规模数据分析的国际合作实验进行了展望。2）我们开发了一个名为“Bricks”的网格模拟器来构建数据网格的性能模型和进行分析。在此基础上，我们对数据网格进行了大量的模拟，并研究了数据网格和GFarm体系结构的特点。我们还开发了一个网格编程环境“Jay”，它是可移植的，适合于对等网格环境3）我们已经开发了一个广泛分布的文件系统，可以避免访问集中在少数节点上的数据网格。我们已经实现了该系统作为一个扩展的GFarm，并检查了系统的有效性。

项目成果

MegaProto: 1 TFlops/10kW Rack Is Feasible Even with Only Commodity Technology

• DOI: 10.1109/sc.2005.45
• 发表时间: 2005-11
• 期刊: ACM/IEEE SC 2005 Conference (SC'05)
• 作者: H. Nakashima;Hiroshi Nakamura;M. Sato;T. Boku;S. Matsuoka;D. Takahashi;Y. Hotta

Gfarm v2: A Grid file system that supports high-performance distributed and parallel data computing

• DOI: 10.5170/cern-2005-002.1172
• 发表时间: 2005
• 作者: O. Tatebe;S. Sekiguchi;Y. Morita;S. Matsuoka;Noriyuki Soda

A Scalable Multi-Replication Framework for Data Grid

数据网格的可扩展多复制框架

• 发表时间: 2005
• 期刊: Proceedings of the 2005 International Symposium on Applications and the Internet
• 作者: Shin' ichiro Takizawa;Yasuhito Takamiya;Hidemoto Nakada;Satoshi Matsuoka
• 通讯作者: Satoshi Matsuoka

GridSpeed : A Web-based Grid Portal Generation Server

GridSpeed：基于 Web 的网格门户生成服务器

• 发表时间: 2004
• 期刊: Proceedings of HPC Asia 2004
• 作者: Toyotaro Suzumura;Hidemoto Nakada;Satoshi Matsuoka;Henri Casanova
• 通讯作者: Henri Casanova

Evaluating Web Services Based Implementations of Grid RPC

评估基于 Web 服务的 Grid RPC 实现

• 发表时间: 2002
• 期刊: Proceedings of 11th IEEE International Symposium on High Performance Distributed Computing (HPDC-11)
• 作者: S.Shirasuna;H.Nakada;S.Matsuoka;S.Sekiguchi
• 通讯作者: S.Sekiguchi