Big Data Remote Access Interfaces for Light Source Science
Big Data Remote Access Interfaces for Light Source Science
复制标题
光源科学大数据远程访问接口
DOI：
10.1109/bdc.2015.37
复制
发表时间：
2015
期刊：
影响因子：
--
通讯作者：
Ian T Foster
中科院分区：
文献类型：
--
作者：
J. Wozniak;K. Chard;B. Blaiszik;Ray Osborn;M. Wilde;Ian T Foster
大数据计算应用程序的效率和可靠性往往取决于它们能否轻松地管理和移动大型分布式数据。例如，在x射线科学中，为了重建、分析、可视化、存储和其他目的，原始数据和各种派生数据都必须在实验大厅、档案馆、超级计算机和用户工作站之间移动。在整个过程中，必须跟踪数据位置，维护原始数据和派生数据之间的关联，数据访问，甚至通过广域网，必须高度响应，以允许交互式可视化。我们在这里使用一个典型的x射线科学工作流来说明最近开发的两种数据移动、归档和标记技术的使用。第一个是可伸缩的目录，用于引用、管理和执行对分布式数据的远程操作。第二个是用于结构化（基于hdf）数据集操作和可视化的新型远程对象接口。我们对这些技术及其在x射线科学问题中的应用的描述，揭示了实验科学中的大数据问题，传统大数据解决方案与科学需求之间的差距，以及弥合这种差距的方法。
The efficiency and reliability of big data computing applications frequently depend on the ease with which they can manage and move large distributed data. For example, in x-ray science, both raw data and various derived data must be moved between experiment halls and archives, supercomputers, and user workstations for reconstruction, analysis, visualization, storage, and other purposes. Throughout, data locations must be tracked and associations between raw and derived data maintained, data accesses, even over wide area networks, must be highly responsive to allow for interactive visualizations. We use here a typical x-ray science workflow to illustrate the use of two recently developed techniques for data movement, archiving, and tagging. The first is a scalable catalog for referencing, managing, and performing remote operations on distributed data. The second is a novel remote object interface for structured (HDF-based) data set manipulation and visualization. Our description of these techniques and their application to x-ray science problems sheds light on big data problems in experimental science, the gap between conventional big data solutions and scientific requirements, and ways in which this gap may be bridged.