Exploiting volatile opportunistic computing resources with Lobster
Exploiting volatile opportunistic computing resources with Lobster
复制标题
使用 Lobster 利用不稳定的机会计算资源
DOI：
--
复制
发表时间：
2015
期刊：
影响因子：
--
通讯作者：
D. Thain
中科院分区：
文献类型：
--
作者：
A. Woodard;M. Wolf;C. Mueller;Benjamín Tovar;P. Donnelly;K. H. Anampa;P. Brenner;K. Lannon;M. Hildreth;D. Thain
在大型强子对撞机（LHC）上使用紧凑μ子螺线管（CMS）进行高能物理实验的分析可能会受到计算资源可用性的限制。作为Notre Dame的计算机科学家和CMS物理学家的共同努力，我们开发了一个机会主义的工作流管理工具Lobster，从大学校园计算池中收集可用的周期。Lobster由管理服务器、文件服务器和工作进程组成，这些进程可以提交给任何可用的计算资源，而不需要root访问权限。Lobster使用工作队列系统来执行任务管理，而CMS特定的软件环境通过CVMFS和Parrot提供。数据通过Chirp和Hadoop进行本地数据存储，通过XrootD访问CMS广域数据联合。开发了一套广泛的监测和诊断工具，以促进系统优化。我们已经使用Notre Dame的20000核集群测试了Lobster，实现了大约8- 10 k个任务同时运行，维持了大约9 Gbit/s的输入数据和340 Mbit/s的输出数据。
Analysis of high energy physics experiments using the Compact Muon Solenoid (CMS) at the Large Hadron Collider (LHC) can be limited by availability of computing resources. As a joint effort involving computer scientists and CMS physicists at Notre Dame, we have developed an opportunistic workflow management tool, Lobster, to harvest available cycles from university campus computing pools. Lobster consists of a management server, file server, and worker processes which can be submitted to any available computing resource without requiring root access. Lobster makes use of the Work Queue system to perform task management, while the CMS specific software environment is provided via CVMFS and Parrot. Data is handled via Chirp and Hadoop for local data storage and XrootD for access to the CMS wide-area data federation. An extensive set of monitoring and diagnostic tools have been developed to facilitate system optimisation. We have tested Lobster using the 20 000-core cluster at Notre Dame, achieving approximately 8-10k tasks running simultaneously, sustaining approximately 9 Gbit/s of input data and 340 Mbit/s of output data.