LuxIO: Intelligent Resource Provisioning and Auto-Configuration for Storage Services
LuxIO: Intelligent Resource Provisioning and Auto-Configuration for Storage Services
复制标题
DOI：
10.1109/hipc56025.2022.00041
复制
发表时间：
2022-12
期刊：
影响因子：
--
通讯作者：
Keith Bateman;N. Rajesh;Jaime Cernuda Garcia;Luke Logan;Jie Ye;Stephen Herbein;Anthony Kougkas;Xian-He Sun
中科院分区：
文献类型：
--
作者：
Keith Bateman;N. Rajesh;Jaime Cernuda Garcia;Luke Logan;Jie Ye;Stephen Herbein;Anthony Kougkas;Xian-He Sun
HPC 中的存储通常是单个远程静态存储 (RSS) 资源。然而，应用程序表现出不同的 I/O 要求，可以通过多存储方法更好地满足这些要求。当前的实践采用在节点本地或共享存储资源上运行的临时存储系统。然而，配置和配置中间存储的负担完全落在用户身上，而全局作业调度程序几乎不提供对自定义部署的支持。缺乏支持通常会导致资源配置过多或不足以及存储系统配置不当。为了缓解这个问题，我们推出了 LuxIO，一种智能存储资源配置和自动配置服务。 LuxIO 构建配置为最匹配 I/O 要求的存储部署。经过 LuxIO 调整的存储服务在常见应用程序和基准测试中显示出高达 2 倍的性能提升，同时在现有作业调度管道之上引入了 93.40 毫秒的最小开销。 LuxIO 在特定工作流程中将资源利用率提高了 25%。
Storage in HPC is typically a single Remote and Static Storage (RSS) resource. However, applications demonstrate diverse I/O requirements that can be better served by a multi-storage approach. Current practice employs ephemeral storage systems running on either node-local or shared storage resources. Yet, the burden of provisioning and configuring intermediate storage falls solely on the users, while global job schedulers offer little to no support for custom deployments. This lack of support often leads to over- or under-provisioning of resources and poorly configured storage systems. To mitigate this, we present LuxIO, an intelligent storage resource provisioning and auto-configuration service. LuxIO constructs storage deployments configured to best match I/O requirements. LuxIO-tuned storage services show performance improvements up to 2× across common applications and benchmarks, while introducing minimal overhead of 93.40 ms on top of existing job scheduling pipelines. LuxIO improves resource utilization by up to 25% in select workflows.