III: Small: Ivory -- A Hadoop Toolkit for Distributed Text Retrieval

文本搜索是一项对现代信息社会至关重要的技术。今天的系统面临着处理以前无法想象的大量文本的艰巨挑战。集群计算是解决规模问题的唯一实用解决方案。该项目利用MapReduce框架(通过开源Hadoop实现)来解决为信息检索应用程序处理大量数据时的健壮性和可伸缩性问题。更广泛地说，我们的目标是探索大型分布式计算环境中处理器、磁盘、内存和网络之间的关系，在这种环境中，单个计算机中的许多假设不再适用。一个相关的例子是Hadoop与实时交互应用程序的需求之间的根本不匹配。由于Hadoop是为面向吞吐量的批处理而设计的，因此目前不提供实时搜索所需的低延迟磁盘访问。分布式内存对象缓存体系结构为解决这一问题提供了一种潜在的解决方案。为了获得更广泛的影响，这项研究的结果将在象牙中实现，象牙是一个基于集群体系结构的开源分布式信息检索工具包。该工具包的提供将有助于维持“云计算”这一新兴领域的活动。有关更多信息，请访问项目网站(http://www.umiacs.umd.edu/~jimmylin/cloud-computing).

Text search is a technology that is vital for modern information-based societies. Today's systems face the daunting challenge of handling quantities of text previously unimaginable. Cluster computing is the only practical solution for addressing the issue of scale. This project leverages the MapReduce framework (via the open-source Hadoop implementation) to tackle issues of robustness and scalability in processing large amounts of data for information retrieval applications. More generally, the goals are to explore the relationship between processor, disk, memory, and network in large distributed computing environments, where many assumptions made in single machines no longer hold. One pertinent example is the fundamental mismatch between Hadoop and the demands of real-time interactive applications. Because it was designed for throughput-oriented batch processing, Hadoop currently does not provide low-latency disk access necessary for real-time search. A distributed in-memory object caching architecture provides a potential solution to this problem.To achieve broader impact, the results of this research will be implemented in Ivory, an open-source toolkit for distributed information retrieval built from the ground up with cluster architectures in mind. The availability of this toolkit will help sustain activities in the emerging area of "cloud computing". Additional information is available on the project website (http://www.umiacs.umd.edu/~jimmylin/cloud-computing).