Collaborative Research: Adaptive Data Parallel Storage

随着研究人员努力提高物理系统模型和模拟的准确性，以及获得更多关于世界的信息，科学应用的I/O需求呈指数级增长。处理计算的这一方面意味着不能忽略I/O差距，而只关注以处理器为中心的问题。作为一个具体的例子，核苷酸数据库Genbank的大小每14个月就会翻一番，目前为33 GB。在商业领域，人们的注意力也转向了信息及其位置。在信息密集型Web应用需求的推动下，越来越多的网络服务正在将计算机系统的重点转移到I/O上。随着数据的日益重要，解决其管理问题的需求也随之而来。不久的将来，由网络连接存储、智能数据设备和工作站以及高性能网络组成的商品集群是一种高度并行的异构型计算机。然而，今天使用的文件抽象还不能充分利用由此产生的计算能力或数据局部性。该奖项研究如何在这样的平台上为一类重要的应用程序提高I/O性能。加州大学圣克鲁斯分校有一个领先的计算生物学小组，拥有100多个节点的Linux集群；他们使用的并行测序代码显然是一个目标。其他团体也表示对这一集群感兴趣；它的使用将是一个极好的应用来源。主要研究目标有三个。首先，开发一个统一计算和存储的I/O编程模型。这将允许程序员表达易于并行化的“存储操作”。其次，构建自适应基础设施并开发分析模型以确定最佳执行路径，该路径可以是并行的，也可以是顺序的。第三，开发离线模型以回答有关如何在此异类环境中分配资源的问题。创建新I/O接口的目标是将计算和数据关联起来，以便代码可以在数据源处轻松执行。例如，假设我们想要在词典中搜索一个单词。标准步骤是打开词典文件，分块读取，在每个块中搜索单词的实例，然后关闭文件。此操作是以处理器为中心的顺序操作；重点是将数据移动到处理器，而不是对数据进行计算。相反，我们建议以新I/O接口的形式将“搜索单词”命令与词典相关联。现在，“搜索单词”变成了上面描述的相同代码的高级抽象，即单个远程过程调用或并行过程(取决于词典的位置)。

The I/O demands of scientific applications are increasing exponentially as researchers strive for greater accuracy in models and simulations of physical systems, and as more information about the world is obtained. To deal with this aspect of computation implies that one cannot ignore the I/O gap and only concentrate on processor-centric issues. As a concrete example, the size of Genbank, a nucleotide database, has been doubling every 14 months and is currently 33 GB. In the commercial arena attention has turned to information and its location also. More and more network services, fueled by the demands of information-intensive Web applications, are shifting the focus in computer systems to I/O.With the growing importance of data comes the need to address its management. The commodity cluster of the near future, which consists of a combination of network-attached storage, intelligent data appliances and workstations, and high-performance networks, is a highly parallel, heterogeneous computer. However, the file abstractions in use today cannot yet fully exploit either the resulting computational power or data locality.This award examines how to improve I/O performance on such a platform for a class of important applications. UC Santa Cruz has a leading computational biology group with a 100+ node Linux cluster; the parallel sequencing codes that they use are an obvious target. Other groups have also expressed interest in this cluster; and its use will be an excellent source of applications. There are three main research objectives. First, develop an I/O programming model that unites computation and storage. This will allow the programmer to express "storage operations" that can be readily parallelized. Second, build an adaptive infrastructure and develop analytic models to determine the optimal execution path, which may be parallel or sequential. Third, develop offline models to answer questions about how to allocate resources within this heterogeneous environment.The goal in creating a new I/O interface is to associate computation and data so that code can be easily executed at the source of the data. For example, suppose we want to search for a word in a dictionary. The standard procedure would be to open the dictionary file, read it in chunks, search for instances of the word in each chunk, and close the file. This operation is sequential and processor-centric; the emphasis is on moving the data to the processor rather than a computation to the data. Instead, we propose to associate a "search for word" command with the dictionary in the form of a new I/O interface. Now the "search for word" becomes a high level abstraction for the same code described above, a single remote procedure call, or a parallel procedure (depending on the location of the dictionary).