BIGDATA: F: DKM: Collaborative Research: PXFS: ParalleX Based Transformative I/O System for Big Data
BIGDATA:F:DKM:协作研究:PXFS:基于 ParalleX 的大数据变革性 I/O 系统
基本信息
- 批准号:1447771
- 负责人:
- 金额:$ 30万
- 依托单位:
- 依托单位国家:美国
- 项目类别:Standard Grant
- 财政年份:2014
- 资助国家:美国
- 起止时间:2014-09-01 至 2017-07-31
- 项目状态:已结题
- 来源:
- 关键词:
项目摘要
Recent decades have seen the development of computational science where modeling and data analysis are critical to exploration, discovery, and refinement of new innovations in science and engineering. More recently the techniques have been applied to arts, social, political and other fields less traditionally reliant on high performance computing. This innovation has grown out of realization some 20 years ago that I/O (input/output) support for high performance parallel and distributed architectures had lagged behind that of pure computational speed, and further that bring I/O up to speed was both critical, and a rather difficult problem. The core hurdle of contemporary I/O on large HPC machines relates to issues of latency in large parts caused by the deficiencies of the historical I/O model that was relevant when computers were exclusively large, centralized, single processor systems shared by many time-sharing programs. In order to improve I/O on scalability on future hardware architectures novel approaches are required.This project is conducting research on an extension of ParalleX, a new highly innovative parallel execution model. The extension provides a powerful I/O interface that allows researchers to create highly efficient data management, discovery, and analysis codes for Big Data applications. This new extension, known as PXFS, is based on HPX, an implementation of ParalleX based on C++, and OrangeFS, a high performance parallel file system. The research goal driving PXFS is to extend HPX objects into I/O space so that the objects become persistent and storage becomes another class of memory, all accessed as a single virtual address space and managed by an event driven dynamic adaptive computation environment. Critical aspects of this approach include futures-based synchronization, dynamic locality management, dynamic resource management, hierarchical name space, and an active global address space (AGAS). The overall goals of PXFS are to eliminate the division of programming imposed by conventional file system through the unification of name spaces and their management, and to minimize global synchronization in order to support asynchronous concurrency. The research methodology is to implement a Map/Reduce application framework using PXFS and evaluate its effectiveness in both performance and ease of use.This project is conducted at three major research universities involving undergraduate and graduate students, post-docs, and high-school teachers and their students. The project includes a PI from the functional genomics field acting as domain science expert in order to focus the development efforts on real world problems. Graduate students and post-docs involved in the project are trained in these areas to promote scientists who understanding both aspects of Big Data problems. The project engages under represented minorities with the goal to inspire them to pursue a career in computer science or genomics. The software developed by the project is available open-source and archived using an integrated source code revision repository, wiki, and bug tracking software system in addition to code releases with accompanying documentation.
近几十年来,计算科学的发展见证了建模和数据分析对于探索、发现和完善科学和工程中的新创新至关重要。最近,这些技术被应用到艺术、社会、政治和其他传统上不那么依赖高性能计算的领域。这一创新源于大约20年前的一种认识,即对高性能并行和分布式体系结构的I/O(输入/输出)支持已经落后于纯计算速度,而且使I/O达到速度既是关键的,也是相当困难的问题。当代大型HPC计算机上的I/O的核心障碍涉及由历史I/O模型的缺陷引起的大部分延迟问题,当计算机完全是由许多分时程序共享的大型、集中式、单处理器系统时,该模型是相关的。为了提高I/O在未来硬件体系结构上的可扩展性,需要新的方法。本项目正在研究一种新的高度创新的并行执行模型Parallex的扩展。该扩展提供了一个强大的I/O接口,使研究人员能够为大数据应用程序创建高效的数据管理、发现和分析代码。这个称为PXFS的新扩展基于HPX和OrangeFS,HPX是基于C++的Parallex实现,OrangeFS是高性能并行文件系统。驱动PXFS的研究目标是将HPX对象扩展到I/O空间,使对象成为持久的,存储成为另一类内存,所有这些都作为单个虚拟地址空间访问,并由事件驱动的动态自适应计算环境管理。该方法的关键方面包括基于未来的同步、动态局部性管理、动态资源管理、分层名称空间和活动全局地址空间(AGAS)。PXFS的总体目标是通过统一命名空间及其管理来消除传统文件系统强加的编程划分,并最小化全局同步以支持异步并发。研究方法是使用PXFS实现Map/Reduce应用程序框架,并从性能和易用性两个方面评估其有效性。该项目在三所主要的研究型大学进行,涉及本科生和研究生、博士后和高中教师及其学生。该项目包括一名来自功能基因组学领域的PI担任领域科学专家,以便将开发努力集中在现实世界的问题上。参与该项目的研究生和博士后都接受了这些领域的培训,以促进了解大数据问题的两个方面的科学家。该项目让有代表性的少数族裔参与进来,目的是激励他们在计算机科学或基因组学领域追求职业生涯。该项目开发的软件是开源的,并使用集成的源代码修订库、wiki和错误跟踪软件系统以及附带文档的代码发布进行存档。
项目成果
期刊论文数量(0)
专著数量(0)
科研奖励数量(0)
会议论文数量(0)
专利数量(0)
数据更新时间:{{ journalArticles.updateTime }}
{{
item.title }}
{{ item.translation_title }}
- DOI:
{{ item.doi }} - 发表时间:
{{ item.publish_year }} - 期刊:
- 影响因子:{{ item.factor }}
- 作者:
{{ item.authors }} - 通讯作者:
{{ item.author }}
数据更新时间:{{ journalArticles.updateTime }}
{{ item.title }}
- 作者:
{{ item.author }}
数据更新时间:{{ monograph.updateTime }}
{{ item.title }}
- 作者:
{{ item.author }}
数据更新时间:{{ sciAawards.updateTime }}
{{ item.title }}
- 作者:
{{ item.author }}
数据更新时间:{{ conferencePapers.updateTime }}
{{ item.title }}
- 作者:
{{ item.author }}
数据更新时间:{{ patent.updateTime }}
Walter Ligon其他文献
Walter Ligon的其他文献
{{
item.title }}
{{ item.translation_title }}
- DOI:
{{ item.doi }} - 发表时间:
{{ item.publish_year }} - 期刊:
- 影响因子:{{ item.factor }}
- 作者:
{{ item.authors }} - 通讯作者:
{{ item.author }}
{{ truncateString('Walter Ligon', 18)}}的其他基金
EAGER: PXFS - A Persistent Storage Model for Extreme Scale
EAGER:PXFS - 超大规模的持久存储模型
- 批准号:
1142905 - 财政年份:2011
- 资助金额:
$ 30万 - 项目类别:
Standard Grant
HECURA: Improving Scalability in Parallel File Systems for High End Computing
HECURA:提高高端计算并行文件系统的可扩展性
- 批准号:
0621441 - 财政年份:2006
- 资助金额:
$ 30万 - 项目类别:
Standard Grant
MRI: Acquisition of a Computational Mini-Grid Supercomputing Facility
MRI:收购计算迷你网格超级计算设施
- 批准号:
0079734 - 财政年份:2000
- 资助金额:
$ 30万 - 项目类别:
Standard Grant
相似海外基金
BIGDATA: F: DKM: Collaborative Research: PXFS: ParalleX Based Transformative I/O System for Big Data
BIGDATA:F:DKM:协作研究:PXFS:基于 ParalleX 的大数据变革性 I/O 系统
- 批准号:
1447650 - 财政年份:2014
- 资助金额:
$ 30万 - 项目类别:
Standard Grant
BIGDATA: F: DKM: DKA: Big Data Modeling and Analysis with Depth and Scale
BIGDATA:F:DKM:DKA:深度和规模的大数据建模和分析
- 批准号:
1447549 - 财政年份:2014
- 资助金额:
$ 30万 - 项目类别:
Standard Grant
BIGDATA: F: DKM: Addressing the two V's of Veracity and Variety in Big Data
BIGDATA:F:DKM:解决大数据中的准确性和多样性这两个 V
- 批准号:
1447795 - 财政年份:2014
- 资助金额:
$ 30万 - 项目类别:
Standard Grant
BIGDATA: F: DKM: Spectral Analysis and Control of Evolving Large Scale Networks
BIGDATA:F:DKM:不断发展的大规模网络的频谱分析和控制
- 批准号:
1447470 - 财政年份:2014
- 资助金额:
$ 30万 - 项目类别:
Standard Grant
BIGDATA: F: DKA: CSD: DKM: Theory and Algorithms for Processing Data with Sparse and Multilinear Structure
BIGDATA:F:DKA:CSD:DKM:稀疏和多线性结构数据处理的理论和算法
- 批准号:
1447879 - 财政年份:2014
- 资助金额:
$ 30万 - 项目类别:
Standard Grant
BIGDATA: F: DKM: Collaborative Research: Making Big Data Active: From Petabytes to Megafolks in Milliseconds
BIGDATA:F:DKM:协作研究:使大数据活跃起来:在毫秒内从 PB 级到百万级数据
- 批准号:
1447720 - 财政年份:2014
- 资助金额:
$ 30万 - 项目类别:
Standard Grant
BIGDATA: F: DKA: DKM: Novel Out-of-core and Parallel Algorithms for Processing Biological Big Data
BIGDATA:F:DKA:DKM:用于处理生物大数据的新型核外并行算法
- 批准号:
1447711 - 财政年份:2014
- 资助金额:
$ 30万 - 项目类别:
Standard Grant
BIGDATA: F: DKM: Plato: A model-based database for compressed spatiotemporal sensor data
BIGDATA:F:DKM:Plato:基于模型的压缩时空传感器数据数据库
- 批准号:
1447943 - 财政年份:2014
- 资助金额:
$ 30万 - 项目类别:
Standard Grant
BIGDATA: F: DKM: Collaborative Research: Making Big Data Active: From Petabytes to Megafolks in Milliseconds
BIGDATA:F:DKM:协作研究:使大数据活跃起来:在毫秒内从 PB 级到百万级数据
- 批准号:
1447826 - 财政年份:2014
- 资助金额:
$ 30万 - 项目类别:
Standard Grant
BIGDATA: F: DKM: Collaborative Research: Scalable Middleware for Managing and Processing Big Data on Next Generation HPC Systems
BIGDATA:F:DKM:协作研究:用于在下一代 HPC 系统上管理和处理大数据的可扩展中间件
- 批准号:
1447861 - 财政年份:2014
- 资助金额:
$ 30万 - 项目类别:
Standard Grant