SciStream: Architecture and Toolkit for Data Streaming between Federated Science Instruments
SciStream: Architecture and Toolkit for Data Streaming between Federated Science Instruments
复制标题
DOI：
10.1145/3502181.3531475
复制
发表时间：
2022-06
期刊：
影响因子：
--
通讯作者：
Joaquín Chung;Wojciech Zacherek;AJ Wisniewski;Zhengchun Liu;Tekin Bicer;R. Kettimuthu;Ian T Foster
中科院分区：
文献类型：
--
作者：
Joaquín Chung;Wojciech Zacherek;AJ Wisniewski;Zhengchun Liu;Tekin Bicer;R. Kettimuthu;Ian T Foster
现代科学仪器，如同步加速器光源上的探测器，以如此高的速率生成数据，因此需要在线处理以进行数据缩减、特征检测、实验指导和其他目的。同样的高数据速率还需要从仪器到远程计算机的内存到内存流，因为本地计算能力有限，并且使用文件系统的数据传输会引入不可接受的延迟。但是，由于科学仪器缺乏直接的外部网络连接，并且由于身份验证和安全要求，在实践中实现高效和安全的内存到内存数据流是具有挑战性的。作为回应，我们在这里提出了SciStream，这是一种基于中间盒的架构，具有控制协议，可以在缺乏直接网络连接的生产者和消费者之间实现高效和安全的内存到内存数据流。我们描述了SciStream用于在生产者和消费者之间建立身份验证和透明连接的协议，并讨论了我们为评估关键SciStream组件的替代实现方法而进行的实验。在变色龙云上的实验表明，与最先进的数据传输方法相比，SciStream将流管道的吞吐量提高了一个数量级，并且与生产者和消费者直接外部连接的理想场景相比，仅增加了~4μsec的延迟。
Modern scientific instruments, such as detectors at synchrotron light sources, generate data at such high rates that online processing is needed for data reduction, feature detection, experiment steering, and other purposes. The same high data rates also demand memory-to-memory streaming from instrument to remote computer, because local computational capacity is limited and data transmissions that engage the file system introduce unacceptable latencies. But efficient and secure memory-to-memory data streaming is challenging to realize in practice, because of a lack of direct external network connectivity for scientific instruments and because of authentication and security requirements. In response, we propose here SciStream, a middlebox-based architecture with control protocols to enable efficient and secure memory-to-memory data streaming between producers and consumers that lack direct network connectivity. We describe the protocols that SciStream uses to establish authenticated and transparent connections between producers and consumers, and we discuss the experiments that we have conducted to evaluate alternative implementation approaches for key SciStream components. Experiments on the Chameleon cloud show that SciStream improves the throughput of a streaming pipeline by an order of magnitude compared with state-of-the-art data transfer methods and adds only ~4μsec latency compared with an ideal scenario in which producers and consumers have direct external connectivity.