Scalable Data Analysis: An Architecture Conscious Approach

技术的进步使个人、组织和政府机构能够收集和存储人类各行各业的海量数据。这里的一个关键挑战是以尽可能高效的方式从这样的万亿级和千万亿级的数据存储中提取可操作的信息，以便领域科学家能够在包括科学、工程、医学和国土安全在内的各个领域取得关键进展。为了实现这一目标，PI寻求在通过高速网络互连的现代集群系统上使用具有体系结构意识的方法来进行可伸缩的数据分析。这项工作的中心论点是，目前用于数据分析的算法经常严重地未充分利用体系结构资源(处理器、内存、磁盘和网络)。该项目试图在科学模拟、生物信息学和安全应用的关键应用驱动因素的背景下解决这一限制。具体地说，局部性增强技术、利用现代体系结构的新功能的能力、有效处理大型核心外数据结构的能力、多级负载平衡和在集群节点之间的工作分配以及支持现代集群上的远程内存分页的机制将在这方面进行研究和利用。这项研究的主要科学成果将包括处理和分析迄今难以处理的大数据集的能力，从而能够在相应的领域中发现新的科学发现，以及能够利用和充分利用基本的并行体系结构来有效地回应和回应领域专家的询问。这项工作的另一个预期结果将是从特定的解决方案中获得部署通用运行时抽象的解决方案，这些抽象可以被大量数据密集型应用程序使用。这项工作的更广泛的结果将是培养有能力的本科生和研究生。将特别鼓励妇女和少数群体参与，并将通过各种举措加强与当地HBCU的现有互动。

Advances in technology have enabled individuals, organizations and government agencies to collect and store massive amounts of data across all walks of human endeavor. A critical challenge here is to extract actionable information from such tera- and peta-scale data stores in as efficient manner as possible so that domain scientists can make critical advances in various fields including the sciences, engineering, medicine and homeland security.Toward this objective, the PI seeks to employ an architecture-conscious approach to scalable data analysis on modern cluster systems interconnected through a high speed network. The central thesis of this work is that current day algorithms for data analysis often grossly under-utilize architectural resources (processors, memory, disk and network). This project seeks to address this limitation in the context of key application drivers drawn from scientific simulations, bioinformatics and security applications. Specifically locality enhancing techniques, the ability to leverage new features of modern architectures, the ability to efficiently work with large out-of-core data structures, multi-level load balancing and distribution of work among cluster nodes and mechanisms that support remote memory paging on modern clusters will be investigated and leveraged in this context. The main scientific outcomes of this research will include the ability to process and analyze hitherto intractably large datasets enabling new scientific discoveries in the corresponding domains and the ability to engage and fully utilize the underlying parallel architecture to respond and react to domain expert queries efficiently. Another expected outcome of this work will be from specific solutions obtained to deploy generic runtime abstractions that can be used by a host of data-intensive applications. The broader outcomes of this work will be to train capable undergraduate and graduate students. Women and minorities will be especially encouraged to participate and existing interactions with a local HBCU will be strengthened through various initiatives.