CSR: Small: Empower Data-Intensive Computing: the integrated data management approach

从计算机系统的角度来看，数字数据有两种：观测数据，即由传感器、监视器、摄像机、文本等电气设备采集的数据；而仿真数据，是通过计算生成的数据。前者代表新出现的互联网数据驱动应用，如社交媒体和数据分析；后者代表了传统的计算驱动应用，如气候建模和计算流体动力学。一般来说，后者需要强一致性才能保证正确性，而前者则不需要。一致性的差异导致了两种文件系统：数据密集型分布式文件系统，以基于mapreduce的Hadoop分布式文件系统（HDFS）为代表；以及以高性能并行文件系统（PFS）为代表的计算密集型文件系统，例如IBM通用并行文件系统（GPFS）。这两种文件系统的设计理念不同，适用于不同的应用程序，并且互不通信。理解海量的收集数据需要强大的计算能力，而大规模的计算需要对大数据的管理。因此，大数据应用需要一个完整的解决方案。本研究开发的综合数据存取系统（IDAS）旨在弥补数据管理的空白。与分布式系统设计中的CAP理论一致，IDAS方法不是被设计成一个新的独立系统，而是作为一个软件层，它提供了一个集成的接口来进行跨平台的数据访问，从HDFS到PFS，或者从PFS到HDFS，读或写，有效且可互换，而不改变用户的应用程序。IDAS的开发计划有三个组成部分：1)建立通信通道，使HDFS和PFS之间可以访问数据；2)设计一个扩展的语义接口，使不同的文件系统可以在不同的计算系统下访问；3)开发优化技术，优化HDFS、PFS和IDAS下的I/O操作。大数据需要数据驱动的互联网计算界和计算驱动的科学计算界共同努力。IDAS为跨平台、跨社区的数据存储、访问和共享服务提供了可持续的、经济高效的基础设施。这项研究将创造先进的解决方案和技术，对提高大规模数据访问和管理的效率有直接影响。由于大数据是科学、工程和工业的国家战略基础设施，拟议的调查将推动广泛的领域。本研究的成功将努力在一个及时、重要、极具挑战性和高影响的问题上取得重大进展，即集成数据访问系统。

From the computer system point of view there are two types of digital data: observational data, the data collected by electrical devices such as sensor, monitor, camera, text, etc.; and simulation data, data generated by computing. The former represents newly emerged internet data-driven applications, such as social media and data analytic; and the latter represents the conventional computing-driven applications, such as climate modeling and computational fluid dynamics. In general, the latter requires strong consistency for correctness and the former does not. The difference in consistency leads to two kinds of file systems: data-intensive distributed file system, represented by the MapReduce-based Hadoop distributed file systems (HDFS); and computing-intensive file systems, represented by the high performance parallel file systems (PFS), such as the IBM general parallel file system (GPFS). These two kinds of file systems are designed with different philosophies, for different applications, and do not talk to each other. Understanding huge amounts of collected data depends on powerful computation, whereas large-scale computation requires the management of large data. Therefore, big data applications demand an integrated solution. The integrated data access system (IDAS) developed under this research is designed to bridge the data management gap.In agreement with the CAP theory in the distributed system design, the IDAS approach is not designed as a new standalone system but as a software layer which provides an integrated interface to conduct cross-platform data access, from HDFS to PFS, or from PFS to HDFS, read or write, effectively and interchangeably without changing the users' applications. The development plan for IDAS has three components: 1) establish the communication channels so that data can be accessed between HDFS and PFS; 2) design an extended semantic interface so that different file systems can be accessed under different computing systems; 3) develop optimization techniques to optimize I/O operation under HDFS, PFS, and under IDAS. Big data requires a joint effort of the data-driven internet computing community and the compute-driven scientific computing community. IDAS provides a sustainable, cost-effective infrastructure for cross-platform, cross-community services of data storage, access, and sharing. This research will create advanced solutions and technologies that will have direct impact on improving the efficiency of data access and management at scale. Since big data is a national strategic infrastructure for science, engineering, and industry, the proposed investigations will advance a broad range of fields. The success of this research will strive to make significant progress of a timely, important, highly challenging, and high-impact problem, namely integrated data access system.