Scalable Data Analysis: An Architecture Conscious Approach

技术的进步使个人、组织和政府机构能够收集和存储人类各行奋进的大量数据。 这里的一个关键挑战是以尽可能有效的方式从这种万亿和千万亿级的数据存储中提取可操作的信息，以便领域科学家可以在包括科学，工程，医学和国土安全在内的各个领域取得重大进展。为了实现这一目标，PI寻求采用一种架构意识的方法来对通过高速网络互连的现代集群系统进行可扩展的数据分析。这项工作的中心论点是，目前的数据分析算法通常严重利用架构资源（处理器，内存，磁盘和网络）。 该项目试图在科学模拟、生物信息学和安全应用的关键应用驱动程序的背景下解决这一限制。特别是局部增强技术，利用现代体系结构的新功能的能力，有效地与大型的核心外数据结构，多层次的负载平衡和集群节点之间的工作分配和机制，支持现代集群上的远程内存分页的能力将在这方面进行研究和利用。这项研究的主要科学成果将包括处理和分析迄今为止难以处理的大型数据集的能力，从而在相应领域实现新的科学发现，以及参与和充分利用底层并行架构来有效地响应和反应领域专家查询的能力。这项工作的另一个预期成果将是获得特定的解决方案，以部署可由大量数据密集型应用程序使用的通用运行时抽象。这项工作的更广泛成果将是培养有能力的本科生和研究生。将特别鼓励妇女和少数民族参与，并将通过各种举措加强与当地HBCU的现有互动。

Advances in technology have enabled individuals, organizations and government agencies to collect and store massive amounts of data across all walks of human endeavor. A critical challenge here is to extract actionable information from such tera- and peta-scale data stores in as efficient manner as possible so that domain scientists can make critical advances in various fields including the sciences, engineering, medicine and homeland security.Toward this objective, the PI seeks to employ an architecture-conscious approach to scalable data analysis on modern cluster systems interconnected through a high speed network. The central thesis of this work is that current day algorithms for data analysis often grossly under-utilize architectural resources (processors, memory, disk and network). This project seeks to address this limitation in the context of key application drivers drawn from scientific simulations, bioinformatics and security applications. Specifically locality enhancing techniques, the ability to leverage new features of modern architectures, the ability to efficiently work with large out-of-core data structures, multi-level load balancing and distribution of work among cluster nodes and mechanisms that support remote memory paging on modern clusters will be investigated and leveraged in this context. The main scientific outcomes of this research will include the ability to process and analyze hitherto intractably large datasets enabling new scientific discoveries in the corresponding domains and the ability to engage and fully utilize the underlying parallel architecture to respond and react to domain expert queries efficiently. Another expected outcome of this work will be from specific solutions obtained to deploy generic runtime abstractions that can be used by a host of data-intensive applications. The broader outcomes of this work will be to train capable undergraduate and graduate students. Women and minorities will be especially encouraged to participate and existing interactions with a local HBCU will be strengthened through various initiatives.