Analytics-driven Efficient Indexing and Query Processing of Extreme Scale AMR Data
Analytics-driven Efficient Indexing and Query Processing of Extreme Scale AMR Data
批准号:
1240682
负责人:
Nagiza Samatova
金额:
$15.0万
依托单位国家:
美国
项目类别:
Standard Grant
财政年份:
2012
资助国家:
美国
项目状态:
已结题
起止时间:
2012-05-01 至 2015-12-31
中文摘要
大质量星星爆炸、飓风运动或聚变能产生等自然现象的高保真科学模拟越来越不仅是计算密集型的,而且是数据密集型的。 由大规模模拟产生的“大”数据需要可持续的数据管理解决方案,以提高能源效率,实现预测性数据分析,并缩短发现时间。为了满足这一关键需求,自适应网格细化(AMR)已成为一种很有前途的技术,可以大幅节省内存、计算和存储资源,同时保持甚至提高仿真准确性。利用AMR,科学家可以自适应地在对模拟现象的“异常”区域进行高分辨率观察和对“通常”区域进行低精度扫视之间进行平衡,AMR对模拟的显著区域进行选择性细粒度细化的优势也正在成为其弱点。其底层网格的非均匀结构导致从数据分析到科学发现的路径沿着对“大”数据的低效模拟后访问。时空模拟的数据生成过程从一个时间步进行到下一个时间步,只需要两个时间步的上下文,同时只在磁盘上存储一个时间步的数据。相比之下,可视化数据分析通常需要可用数据的完整上下文,而不仅仅是单个时间步。事实上,由局部时空关系驱动的模拟在很大程度上是为了通过交互式的“假设”数据探索来发现或解释非局部和大规模的时空关系。因此,数据上下文的根本差异和访问模式的异质性需要分析驱动的科学数据管理解决方案。为了弥补这一差距,本项目旨在通过将数据分析和数据简化作为数据库设计和查询处理的头等公民,显著推进“大”AMR数据的数据库管理。为了支持分析驱动的高效查询处理,它提供了从传统的数据索引到关于数据压缩和存储中的分层数据布局的信息的拟议索引的变革性转变。前者会产生大量的存储开销,而后者预计将是存储友好的,响应用户查询的数据检索速度明显加快。 为了实现这一雄心勃勃的目标,在数据密集型计算机科学的三个广泛领域提出了创新:语义建模,科学数据库管理和高性能存储。 具体而言,科学数据库管理将认识到AMR数据特有的丰富数据访问和查询语义,能够在模拟数据生成期间进行原位数据压缩和索引,并通过新颖的数据布局方法针对异构访问模式进行优化。这项研究是与三个国家实验室的研究团队密切合作进行的,他们在高性能存储、AMR和高保真气候变化模拟方面拥有开创性的技术。 本研究中开发的方法和形式主义预计将适用于广泛的科学和工程问题,这些问题利用模拟模型通过“大”数据分析预测理解物理过程。 在该项目下开发的技术预计将加速科学工作流程中数据驱动的探索和知识发现的关键过程。该项目还计划在“大”数据教育、多样性、社区参与以及通过学术出版物和开源软件传播成果方面做出重大努力。
英文摘要
High-fidelity scientific simulations of natural phenomena, such as massive star explosion, hurricane movement, or fusion energy production increasingly become not only compute-intensive but also data-intensive. "Big" data produced by large-scale simulations calls for sustainable data management solutions that increase energy efficiency, enable predictive data analytics, and reduce time-to-discovery. In response to this critical need, Adaptive Mesh Refinement (AMR) has emerged as a promising technology to achieve considerable savings in memory, computation, and storage resources, while maintaining or even increasing simulation accuracy. With AMR, scientists can adaptively balance between high-resolution look into "the unusual" areas and low-precision glance at "the usual" areas of the simulated phenomena.The strength of AMR selective fine-grain refinement of salient areas of the simulationis also becoming its weakness. The non-uniform structure of its underlying mesh leads to inefficient post-simulation access to the "Big" data along the path from data analytics to scientific discovery. The data generation process of space-time simulation proceeds from one time step to the next and requires the context of only two time steps, while storing data for only one time step on the disk. In contrast, visual data analytics often requires the full context of the available data, not just a single time step. In fact, simulations that are driven by local space-time relationships are largely performed with the purpose of discovering or explaining non-local and large-scale space-time relationships through interactive "what-if" data exploration. Thus, the fundamental differences in data context and heterogeneity of access patterns call for analytics-driven scientific data management solutions.To close this gap, this project aims to significantly advance database management for "Big" AMR data by making data analytics and data reduction the first class citizens of the database design and query processing. To support analytics-driven efficient query processing, it offers a transformative shift from the traditional indexing of data to the proposed indexing of information about data compression and hierarchical data layout in storage. The former incurs substantial storage overhead, while the latter is anticipated to be storage-friendly with significantly faster data retrieval in response to user queries. To realize this ambitious goal, innovations are proposed in three broad areas of data-intensive computer science: semantic modeling, scientific database management, and high performance storage. Specifically, the scientific database management is going to be cognizant of rich data access-and-query semantics specific to AMR data, capable of in situ data compression and indexing during data generation by the simulation, and optimized for heterogeneous access patterns through novel data layout methodologies. This research is being conducted in a close partnership with the three national labs' research teams with pioneering technologies in high performance storage, AMR, and high-fidelity climate change simulations. The approaches and formalisms developed in this research are expected to be applicable to a broad range of scientific and engineering problems that utilize simulation models for predictive understanding of physical processes through "Big" data analytics. The techniques developed under this project are expected to accelerate the crucial process of data-driven exploration and knowledge discovery in the scientific workflow. This project also plans significant efforts in "Big" data education, diversity, community engagement, and the dissemination of results through academic publications and open-source software.
期刊论文(0)
专著(0)
科研奖励(0)
会议论文
Collaborative Research: Understanding Climate Change: A Data Driven Approach
-
批准号:1028746
-
项目类别:Continuing Grant
-
资助金额:$179.97万
-
财政年份:2010
-
负责人:Nagiza Samatova
-
依托单位:
Workshop on Mathematics for Petascale Data, June 3-5, 2008, Rockville, MD
-
批准号:0829830
-
项目类别:Standard Grant
-
资助金额:$2.0万
-
财政年份:2008
-
负责人:Nagiza Samatova
-
依托单位:
国内基金
海外基金
Data-driven Recommendation System Construction of an Online Medical Platform Based on the Fusion of Information
-
批准号:--
-
项目类别:外国青年学者研究基金项目
-
资助金额:--
-
批准年份:2024
-
负责人:江洋子
-
依托单位:
基于Cache的远程计时攻击研究
-
批准号:60772082
-
项目类别:面上项目
-
资助金额:28.0万元
-
批准年份:2007
-
负责人:王韬
-
依托单位: