Toward scalable monitoring on large-scale storage for software defined cyberinfrastructure
Toward scalable monitoring on large-scale storage for software defined cyberinfrastructure
复制标题
DOI：
10.1145/3149393.3149402
复制
发表时间：
2017-11
期刊：
影响因子：
--
通讯作者：
A. Paul;S. Tuecke;Ryan Chard;A. Butt;K. Chard;Ian T Foster
中科院分区：
文献类型：
--
作者：
A. Paul;S. Tuecke;Ryan Chard;A. Butt;K. Chard;Ian T Foster
随着研究过程变得更加协作和日益面向数据，需要新的技术来有效地管理和自动化数据生命周期的关键但繁琐的方面。研究人员现在花费大量时间复制、编目、共享、分析和清除分布在巨大存储网络上的大量数据。软件定义的网络基础架构(SDCI)通过增强现有存储系统来支持基于高级数据管理策略规范的操作的自动执行，从而为此问题提供了解决方案。我们的SDCI实施称为Ripple，它依赖于部署在存储资源上的代理来检测数据事件并对其采取行动。然而，当前的监控技术(如intify)通常不适用于大型或并行文件系统(如Lustre)。我们在这里描述了一种在大型(多PB)Lustre文件系统上进行可扩展、轻量级事件检测的方法。Ripple和Lustre显示器共同实现了跨个人设备和领导力计算平台的新型生命周期自动化。
As research processes become yet more collaborative and increasingly data-oriented, new techniques are needed to efficiently manage and automate the crucial, yet tedious, aspects of the data life-cycle. Researchers now spend considerable time replicating, cataloging, sharing, analyzing, and purging large amounts of data, distributed over vast storage networks. Software Defined Cyberinfrastructure (SDCI) provides a solution to this problem by enhancing existing storage systems to enable the automated execution of actions based on the specification of high-level data management policies. Our SDCI implementation, called Ripple, relies on agents being deployed on storage resources to detect and act on data events. However, current monitoring technologies, such as inotify, are not generally available on large or parallel file systems, such as Lustre. We describe here an approach for scalable, lightweight, event detection on large (multi-petabyte) Lustre file systems. Together, Ripple and the Lustre monitor enable new types of lifecycle automation across both personal devices and leadership computing platforms.