A Job Sizing Strategy for High-Throughput Scientific Workflows
A Job Sizing Strategy for High-Throughput Scientific Workflows
复制标题
高通量科学工作流程的工作规模策略
DOI：
--
复制
发表时间：
2018
影响因子：
5.3
通讯作者：
M. Livny
中科院分区：
文献类型：
--
作者：
Benjamín Tovar;Rafael Ferreira da Silva;G. Juve;E. Deelman;W. Allcock;D. Thain;M. Livny
计算设施的用户在提交作业以供执行时必须做出关键决策：每个作业应该请求多少资源(如内核、内存和磁盘)？如果请求太小，作业可能会因为资源耗尽而失败；如果请求太大，作业可能会成功，但会浪费资源。当在高吞吐量工作流中运行数十万个作业时，此决策尤其重要，因为高吞吐量工作流可能会显示资源消耗的复杂、长尾分布。在本文中，我们提出了一种解决作业大小问题的策略：(1)在应用程序运行时在用户空间对其进行监控和测量；(2)将资源使用情况收集到在线归档中；(3)根据历史数据自动调整作业大小，以最大化吞吐量或减少浪费。我们对解决方案进行了分析评估，并提供了将该技术应用于包含数十万个工作岗位的高通量物理和生物信息学工作流的案例研究，表明与幼稚的方法相比，吞吐量提高了10%-400%。
The user of a computing facility must make a critical decision when submitting jobs for execution: how many resources (such as cores, memory, and disk) should be requested for each job? If the request is too small, the job may fail due to resource exhaustion; if the request is too large, the job may succeed, but resources will be wasted. This decision is especially important when running hundreds of thousands of jobs in a high throughput workflow, which may exhibit complex, long tailed distributions of resource consumption. In this paper, we present a strategy for solving the job sizing problem: (1) applications are monitored and measured in user-space as they run; (2) the resource usage is collected into an online archive; and (3) jobs are automatically sized according to historical data in order to maximize throughput or minimize waste. We evaluate the solution analytically, and present case studies of applying the technique to high throughput physics and bioinformatics workflows consisting of hundreds of thousands of jobs, demonstrating an increase in throughput of 10-400 percent compared to naive approaches.
XSEDE HPC 系统的全面作业级别资源使用情况测量和分析
DOI：
10.1145/2484762.2484781
发表时间：
2013
期刊：
Proceedings of the Conference on Extreme Science and Engineering Discovery Environment: Gateway to Discovery (XSEDE '13
影响因子：
--
作者：
Lu, Charng-Da;Browne, James;DeLeon, Robert L.;Hammond, John;Barth, William;Furlani, Thomas R.;Gallo, Steven M.;Jones, Matthew D.;Patra, Abani K.
通讯作者：
Patra, Abani K.