Solving the Container Explosion Problem for Distributed High Throughput Computing
Solving the Container Explosion Problem for Distributed High Throughput Computing
复制标题
解决分布式高吞吐量计算的容器爆炸问题
DOI：
10.1109/ipdps47924.2020.00048
复制
发表时间：
2020
期刊：
影响因子：
--
通讯作者：
Thain, Douglas
中科院分区：
文献类型：
--
作者：
Shaffer, Tim;Hazekamp, Nicholas;Blomer, Jakob;Thain, Douglas
容器技术在先进的计算设施中得到了更广泛的使用，用于管理必须在多个站点执行的高度复杂的应用程序。然而，在分布式高吞吐量计算环境中，容器的无限制使用可能导致容器爆炸问题。如果为分派到站点的作业的每个变体生成新的容器映像，则共享存储很快就会被超过。另一方面，如果使用单个大型容器映像来满足多种需求，则该容器的大小可能成为存储和运输的问题。为了解决这个问题，我们观察到许多容器都有一个由结构化包管理器生成的内部结构，这些信息可以用于战略性地联合收割机和共享容器映像。我们开发LANDLORD利用这一属性，并通过模拟研究和高能物理应用的经验测量相结合来评估其性能。
Container technologies are seeing wider use at advanced computing facilities for managing highly complex applications that must execute at multiple sites. However, in a distributed high throughput computing setting, the unrestricted use of containers can result in the container explosion problem. If a new container image is generated for each variation of a job dispatched to a site, shared storage is soon exceeded. On the other hand, if a single large container image is used to meet multiple needs, the size of that container may become a problem for storage and transport. To address this problem, we observe that many containers have an internal structure generated by a structured package manager, and this information could be used to strategically combine and share container images. We develop LANDLORD to exploit this property and evaluate its performance through a combination of simulation studies and empirical measurement of high energy physics applications.
DOI：
--
发表时间：
2015
期刊：
Computing in science & engineering (Print)
影响因子：
--
作者：
J. Blomer;P. Buncic;René Meusel;G. Ganis;I. Sfiligoi;D. Thain
通讯作者：
D. Thain