Two stage cluster for resource optimization with Apache Mesos
Two stage cluster for resource optimization with Apache Mesos
复制标题
DOI：
--
复制
发表时间：
2019-05
期刊：
影响因子：
--
通讯作者：
Gourav Rattihalli;Pankaj Saha;M. Govindaraju;Devesh Tiwari
中科院分区：
文献类型：
--
作者：
Gourav Rattihalli;Pankaj Saha;M. Govindaraju;Devesh Tiwari
由于作业的资源估计是困难的，用户经常高估他们的需求。由于用户提供的作业资源估计不准确，商业云和学术校园集群都存在资源利用率低和等待时间长的问题。我们提出了一种在小集群中作业在大集群中运行前统计估计其实际资源需求的方法。对小集群的初始估计可以让我们了解一个作业实际需要多少资源。这个初始估计使我们能够准确地为队列中的待处理作业分配资源，从而提高吞吐量和资源利用率。在我们的实验中，我们确定的资源利用率估计的内存和CPU的平均准确率分别为90%和94%，而与Apache Aurora和Apache Mesos上的默认作业提交方法相比，我们的内存和CPU的平均利用率分别提高了22%和53%。
As resource estimation for jobs is difficult, users often overestimate their requirements. Both commercial clouds and academic campus clusters suffer from low resource utilization and long wait times as the resource estimates for jobs, provided by users, is inaccurate. We present an approach to statistically estimate the actual resource requirement of a job in a Little cluster before the run in a Big cluster. The initial estimation on the little cluster gives us a view of how much actual resources a job requires. This initial estimate allows us to accurately allocate resources for the pending jobs in the queue and thereby improve throughput and resource utilization. In our experiments, we determined resource utilization estimates with an average accuracy of 90% for memory and 94% for CPU, while we make better utilization of memory by an average of 22% and CPU by 53%, compared to the default job submission methods on Apache Aurora and Apache Mesos.