KDI: Accessing Large Distributed Archives in Astronomy and Particle Physics

9980044 Szalay本提案要求提供资金，以支持对天文学和粒子物理学中现在或即将获得的超大数据集的各种组织和访问方法的研究。这类实验的典型数据采集速率超过每年几太字节，在不久的将来将达到每年拍字节的速率。如此大量的数据构成了不小的挑战，尤其是因为必须通过地理上分散的大型协作快速、方便地访问这些数据。天文学数据包括(1)斯隆数字天空调查(SDSS)，一张40 TB的北部天空数字地图；(2)来自GALEX卫星的数据(约.20 TB)；以及(3)来自航天飞机AMS实验的数据。在粒子物理方面，瑞士日内瓦CERN实验室大型强子对撞机(LHC)的CMS实验数据每年将大大超过1PB。这项研究将解决以下问题：(1)组织数据以实现有效访问。这将需要探索几种高级数据结构，以便可以预测未来查询的方式存储数据。(2)数据的广泛分布存储。这就需要从网络和分布式计算系统的角度提出一种新的方法。提出者将开发一个查询代理中间件来执行在各种数据位置上的查询。(3)高效处理整个重大档案。这需要开发并行处理工具，这种工具可以利用商用处理器和磁盘的阵列来实现每小时几TB的分析。9980044 Szalay本提案要求提供资金，以支持对天文学和粒子物理学中现在或即将获得的超大数据集的各种组织和访问方法的研究。这类实验的典型数据采集速率超过每年几太字节，在不久的将来将达到每年拍字节的速率。如此大量的数据构成了不小的挑战，尤其是因为必须通过地理上分散的大型协作快速、方便地访问这些数据。天文学数据包括(1)斯隆数字天空调查(SDSS)，一张40 TB的北部天空数字地图；(2)来自GALEX卫星的数据(约.20 TB)；以及(3)来自航天飞机AMS实验的数据。在粒子物理方面，瑞士日内瓦CERN实验室大型强子对撞机(LHC)的CMS实验数据每年将大大超过1PB。这项研究将解决以下问题：(1)组织数据以实现有效访问。这将需要探索几种高级数据结构，以便可以预测未来查询的方式存储数据。(2)数据的广泛分布存储。这就需要从网络和分布式计算系统的角度提出一种新的方法。提出者将开发一个查询代理中间件来执行在各种数据位置上的查询。(3)高效处理整个重大档案。这需要开发并行处理工具，这种工具可以利用商用处理器和磁盘的阵列来实现每小时几TB的分析。9980044 Szalay本提案要求提供资金，以支持对天文学和粒子物理学中现在或即将获得的超大数据集的各种组织和访问方法的研究。这类实验的典型数据采集速率超过每年几太字节，在不久的将来将达到每年拍字节的速率。如此大量的数据构成了不小的挑战，尤其是因为必须通过地理上分散的大型协作快速、方便地访问这些数据。天文学数据包括(1)斯隆数字天空调查(SDSS)，一张40 TB的北部天空数字地图；(2)来自GALEX卫星的数据(约.20 TB)；以及(3)来自航天飞机AMS实验的数据。在粒子物理方面，瑞士日内瓦CERN实验室大型强子对撞机(LHC)的CMS实验数据每年将大大超过1PB。这项研究将解决以下问题：(1)组织数据以实现有效访问。这将需要探索几种高级数据结构，以便可以预测未来查询的方式存储数据。(2)数据的广泛分布存储。这就需要从网络和分布式计算系统的角度提出一种新的方法。提出者将开发一个查询代理中间件来执行在各种数据位置上的查询。(3)高效处理整个重大档案。这需要开发并行处理工具，这种工具可以利用商用处理器和磁盘的阵列来实现每小时几TB的分析。

9980044SzalayThis proposal requests funding to support research into various ways of organizing and accessing the exceedingly large data sets now or soon to be acquired in astronomy and particle physics. The typical data acquisition rate for such experiments exceeds several Terabytes per year, and in the very near future will reach the Petabyte per year rate. Such amounts of data pose non-trivial challenges, in particular since they must be accessed quickly and conveniently by a large and geographically dispersed collaboration. The astronomy data includes (1) the Sloan Digital Sky Survey (SDSS), a 40 TB digital map of the northern sky; (2) data from the GALEX satellite (approx. 20 TB); and (3) data from the space shuttle based AMS experiment. In particle physics, data from the CMS experiment at the Large Hadron Collider (LHC) at the CERN laboratory in Geneva Switzerland will significantly exceed 1 PB per year.This research will address the following issues: (1) Organization of the data for efficient access. This will require exploration of several advanced data structures so that data may be stored in a fashion which anticipates future queries. (2) Storage of the data in widely distributed locations. This requires a novel approach from the perspective of both networking and distributed computing systems. Proposers will develop a middleware of Query Agents to perform the queries at the verious data locations. (3) Efficient handling of entire major archives. This requires development of parallel processing tools that can exploit arrays of commodity processors and disks to allow analysis of several Terabytes per hour. 9980044SzalayThis proposal requests funding to support research into various ways of organizing and accessing the exceedingly large data sets now or soon to be acquired in astronomy and particle physics. The typical data acquisition rate for such experiments exceeds several Terabytes per year, and in the very near future will reach the Petabyte per year rate. Such amounts of data pose non-trivial challenges, in particular since they must be accessed quickly and conveniently by a large and geographically dispersed collaboration. The astronomy data includes (1) the Sloan Digital Sky Survey (SDSS), a 40 TB digital map of the northern sky; (2) data from the GALEX satellite (approx. 20 TB); and (3) data from the space shuttle based AMS experiment. In particle physics, data from the CMS experiment at the Large Hadron Collider (LHC) at the CERN laboratory in Geneva Switzerland will significantly exceed 1 PB per year.This research will address the following issues: (1) Organization of the data for efficient access. This will require exploration of several advanced data structures so that data may be stored in a fashion which anticipates future queries. (2) Storage of the data in widely distributed locations. This requires a novel approach from the perspective of both networking and distributed computing systems. Proposers will develop a middleware of Query Agents to perform the queries at the verious data locations. (3) Efficient handling of entire major archives. This requires development of parallel processing tools that can exploit arrays of commodity processors and disks to allow analysis of several Terabytes per hour. 9980044SzalayThis proposal requests funding to support research into various ways of organizing and accessing the exceedingly large data sets now or soon to be acquired in astronomy and particle physics. The typical data acquisition rate for such experiments exceeds several Terabytes per year, and in the very near future will reach the Petabyte per year rate. Such amounts of data pose non-trivial challenges, in particular since they must be accessed quickly and conveniently by a large and geographically dispersed collaboration. The astronomy data includes (1) the Sloan Digital Sky Survey (SDSS), a 40 TB digital map of the northern sky; (2) data from the GALEX satellite (approx. 20 TB); and (3) data from the space shuttle based AMS experiment. In particle physics, data from the CMS experiment at the Large Hadron Collider (LHC) at the CERN laboratory in Geneva Switzerland will significantly exceed 1 PB per year.This research will address the following issues: (1) Organization of the data for efficient access. This will require exploration of several advanced data structures so that data may be stored in a fashion which anticipates future queries. (2) Storage of the data in widely distributed locations. This requires a novel approach from the perspective of both networking and distributed computing systems. Proposers will develop a middleware of Query Agents to perform the queries at the verious data locations. (3) Efficient handling of entire major archives. This requires development of parallel processing tools that can exploit arrays of commodity processors and disks to allow analysis of several Terabytes per hour.