POSH: A Data-Aware Shell
POSH: A Data-Aware Shell
复制标题
DOI：
--
复制
发表时间：
2020
期刊：
影响因子：
--
通讯作者：
Deepti Raghavan;Sadjad Fouladi;P. Levis;M. Zaharia
中科院分区：
文献类型：
--
作者：
Deepti Raghavan;Sadjad Fouladi;P. Levis;M. Zaharia
我们提出了P奥什，这是一个使用I/O繁重的组件加速shell应用程序的框架，例如使用命令行实用程序进行数据分析。远程存储（如网络文件系统）会严重限制这些应用程序的性能：数据在网络上往返，而客户端的计算量相对较小。通过将代码移动到数据来减少数据移动可以提高性能。P奥什通过将I/O密集型部分重新加载到更靠近数据的代理服务器，自动优化在远程存储上运行的未经修改的艾德I/O密集型shell应用程序。代理可以直接在存储服务器上运行，也可以在比客户机更靠近存储层的机器上运行。P奥什拦截shell管道，并使用称为注释的元数据来决定在管道中的何处运行每个命令。我们解决出现的三个主要挑战：一个注释语言，允许P奥什了解哪些文件的命令将访问，调度算法，将命令，以尽量减少数据移动，和系统运行时执行分布式调度，但保留本地语义。我们在真实的shell管道上对P奥什进行了基准测试，例如图像处理，网络安全分析，日志分析，分布式系统调试和git。我们发现，与NFS相比，P奥什提供了1.6倍到15倍的加速比，而无需对应用程序进行任何修改。
We present P OSH , a framework that accelerates shell applications withI/O-heavycomponents,suchas data analytics with command-line utilities. Remote storage such as networked filesystems can severely limit the performance of these applications: data makes a round trip over the network for relatively little computation at the client. Reducing the data movement by moving the code to the data can improve performance. P OSH automatically optimizes unmodified I/O-intensive shell applications running over remote storage by offloading the I/O-intensive portions to proxy servers closer to the data. A proxy can run directly on a storage server, or on a machine closer to the storage layer than the client. P OSH intercepts shell pipelines and uses metadata called annotations to decide where to run each command within the pipeline. We address three principal challenges that arise: an annotation language that allows P OSH to understand which files a command will access, a scheduling algorithm that places commands to minimize data movement, and a system runtime to execute a distributed schedule but retain local semantics. We benchmark P OSH on real shell pipelines such as image processing, network security analysis, log analysis, distributed system debugging, and git . We find that P OSH provides speedups ranging from 1.6 × to 15 × compared to NFS, without requiring any modifications to the applications.