Scaling Data Intensive Physics Applications to 10k Cores on Non-dedicated Clusters with Lobster
Scaling Data Intensive Physics Applications to 10k Cores on Non-dedicated Clusters with Lobster
复制标题
使用 Lobster 将数据密集型物理应用扩展到非专用集群上的 10k 核心
DOI：
10.1109/cluster.2015.53
复制
发表时间：
2015
期刊：
影响因子：
--
通讯作者：
M. Hildreth
中科院分区：
文献类型：
--
作者：
A. Woodard;M. Wolf;C. Müller;N. Valls;Benjamín Tovar;P. Donnelly;Peter Ivie;K. H. Anampa;P. Brenner;D. Thain;K. Lannon;M. Hildreth
高能物理（HEP）社区依靠全球计算和数据中心网络来分析大型强子对撞机（LHC）多次实验产生的数据。然而，这个全球网络并不能满足所有的研究需求。雄心勃勃的研究人员通常希望利用没有集成到全球网络中的计算资源，包括私有集群、商业云和其他生产网格。为了实现这些用例，我们构建了Lobster，这是一个用于在非专用集群上部署数据密集型高吞吐量应用程序的系统。这需要解决与非专用资源相关的多个问题，包括工作分解、软件交付、并发管理、数据访问、数据合并和性能故障排除。通过这些技术，我们演示了Lobster在10k核上有效运行，产生的吞吐量可与LHC基础设施中一些最大的专用集群相媲美。
The high energy physics (HEP) community relies upon a global network of computing and data centers to analyze data produced by multiple experiments at the Large Hadron Collider (LHC). However, this global network does not satisfy all research needs. Ambitious researchers often wish to harness computing resources that are not integrated into the global network, including private clusters, commercial clouds, and other production grids. To enable these use cases, we have constructed Lobster, a system for deploying data intensive high throughput applications on non-dedicated clusters. This requires solving multiple problems related to non-dedicated resources, including work decomposition, software delivery, concurrency management, data access, data merging, and performance troubleshooting. With these techniques, we demonstrate Lobster running effectively on 10k cores, producing throughput at a level comparable with some of the largest dedicated clusters in the LHC infrastructure.
DOI：
10.1088/1742-6596/664/6/062031
发表时间：
2015-12
期刊：
Journal of Physics: Conference Series
影响因子：
--
作者：
J. Balcas;S. Belforte;B. Bockelman;D. Colling;O. Gutsche;D. Hufnagel;F. Khan;K. Larson;J. Letts;M. Mascheroni;D. Mason;A. McCrea;S. Piperov;M. Saiz-Santos;I. Sfiligoi;A. Tanasijczuk;C. Wissing
通讯作者：
J. Balcas;S. Belforte;B. Bockelman;D. Colling;O. Gutsche;D. Hufnagel;F. Khan;K. Larson;J. Letts;M. Mascheroni;D. Mason;A. McCrea;S. Piperov;M. Saiz-Santos;I. Sfiligoi;A. Tanasijczuk;C. Wissing