CSR: Small: Empower Data-Intensive Computing: the integrated data management approach

CSR：小：赋能数据密集型计算：集成数据管理方法

• 批准号: 1526887
• 负责人: Xian-He Sun
• 金额: $ 40万
• 依托单位: Illinois Institute of Technology
• 依托单位国家: 美国
• 项目类别: Standard Grant
• 财政年份: 2015
• 资助国家: 美国
• 起止时间: 2015-09-01 至 2019-08-31
• 项目状态: 已结题
• 来源: https://www.nsf.gov/awardsearch/showAward?AWD_ID=1526887&HistoricalAwards=false
• 关键词: CSR; Small; Empower; Data; Intensive

From the computer system point of view there are two types of digital data: observational data, the data collected by electrical devices such as sensor, monitor, camera, text, etc.; and simulation data, data generated by computing. The former represents newly emerged internet data-driven applications, such as social media and data analytic; and the latter represents the conventional computing-driven applications, such as climate modeling and computational fluid dynamics. In general, the latter requires strong consistency for correctness and the former does not. The difference in consistency leads to two kinds of file systems: data-intensive distributed file system, represented by the MapReduce-based Hadoop distributed file systems (HDFS); and computing-intensive file systems, represented by the high performance parallel file systems (PFS), such as the IBM general parallel file system (GPFS). These two kinds of file systems are designed with different philosophies, for different applications, and do not talk to each other. Understanding huge amounts of collected data depends on powerful computation, whereas large-scale computation requires the management of large data. Therefore, big data applications demand an integrated solution. The integrated data access system (IDAS) developed under this research is designed to bridge the data management gap.In agreement with the CAP theory in the distributed system design, the IDAS approach is not designed as a new standalone system but as a software layer which provides an integrated interface to conduct cross-platform data access, from HDFS to PFS, or from PFS to HDFS, read or write, effectively and interchangeably without changing the users' applications. The development plan for IDAS has three components: 1) establish the communication channels so that data can be accessed between HDFS and PFS; 2) design an extended semantic interface so that different file systems can be accessed under different computing systems; 3) develop optimization techniques to optimize I/O operation under HDFS, PFS, and under IDAS. Big data requires a joint effort of the data-driven internet computing community and the compute-driven scientific computing community. IDAS provides a sustainable, cost-effective infrastructure for cross-platform, cross-community services of data storage, access, and sharing. This research will create advanced solutions and technologies that will have direct impact on improving the efficiency of data access and management at scale. Since big data is a national strategic infrastructure for science, engineering, and industry, the proposed investigations will advance a broad range of fields. The success of this research will strive to make significant progress of a timely, important, highly challenging, and high-impact problem, namely integrated data access system.

从计算机系统的角度来看，数字数据有两种类型：观测数据，由传感器、监视器、摄像机、文本等电子设备采集的数据;和模拟数据，通过计算产生的数据。前者代表新兴的互联网数据驱动应用，如社交媒体和数据分析;后者代表传统的计算驱动应用，如气候建模和计算流体动力学。一般来说，后者需要强一致性的正确性，而前者不需要。一致性的差异导致了两种文件系统：数据密集型分布式文件系统，以基于MapReduce的Hadoop分布式文件系统（HDFS）为代表;计算密集型文件系统，以高性能并行文件系统（PFS）为代表，如IBM通用并行文件系统（GPFS）。这两种文件系统是根据不同的原理设计的，用于不同的应用程序，并且彼此不通信。理解大量收集的数据依赖于强大的计算，而大规模计算需要管理大量数据。因此，大数据应用需要集成解决方案。本研究所开发的集成数据访问系统（IDAS）旨在弥补数据管理的不足，与分布式系统设计中的CAP理论相一致，IDAS方法不是设计成一个新的独立系统，而是作为一个软件层，提供一个集成接口来进行跨平台的数据访问，从HDFS到PFS，或从PFS到HDFS，读或写，有效地和可互换地，而不改变用户的应用。IDAS的开发计划有三个组成部分：1）建立通信通道，使数据可以在HDFS和PFS之间访问; 2）设计一个扩展的语义接口，使不同的文件系统可以在不同的计算系统下访问; 3）开发优化技术，以优化HDFS，PFS和IDAS下的I/O操作。大数据需要数据驱动的互联网计算社区和计算驱动的科学计算社区的共同努力。IDAS为跨平台、跨社区的数据存储、访问和共享服务提供了一个可持续的、具有成本效益的基础设施。这项研究将创建先进的解决方案和技术，对提高大规模数据访问和管理的效率产生直接影响。由于大数据是科学，工程和工业的国家战略基础设施，因此拟议的调查将推动广泛的领域。本研究的成功将力争使一个及时、重要、极具挑战性、影响力大的问题，即集成数据访问系统取得重大进展。