III: Small: Ivory -- A Hadoop Toolkit for Distributed Text Retrieval

III：小：Ivory——分布式文本检索的 Hadoop 工具包

• 批准号: 0916043
• 负责人: Jimmy Lin
• 金额: $ 49.97万
• 依托单位: University of Maryland, College Park
• 依托单位国家: 美国
• 项目类别: Standard Grant
• 财政年份: 2009
• 资助国家: 美国
• 起止时间: 2009-10-01 至 2013-09-30
• 项目状态: 已结题
• 来源: https://www.nsf.gov/awardsearch/showAward?AWD_ID=0916043&HistoricalAwards=false
• 关键词: III; Small; Ivory; Hadoop; Toolkit

Text search is a technology that is vital for modern information-based societies. Today's systems face the daunting challenge of handling quantities of text previously unimaginable. Cluster computing is the only practical solution for addressing the issue of scale. This project leverages the MapReduce framework (via the open-source Hadoop implementation) to tackle issues of robustness and scalability in processing large amounts of data for information retrieval applications. More generally, the goals are to explore the relationship between processor, disk, memory, and network in large distributed computing environments, where many assumptions made in single machines no longer hold. One pertinent example is the fundamental mismatch between Hadoop and the demands of real-time interactive applications. Because it was designed for throughput-oriented batch processing, Hadoop currently does not provide low-latency disk access necessary for real-time search. A distributed in-memory object caching architecture provides a potential solution to this problem.To achieve broader impact, the results of this research will be implemented in Ivory, an open-source toolkit for distributed information retrieval built from the ground up with cluster architectures in mind. The availability of this toolkit will help sustain activities in the emerging area of "cloud computing". Additional information is available on the project website (http://www.umiacs.umd.edu/~jimmylin/cloud-computing).

文本搜索是现代信息社会的一项重要技术。今天的系统面临着处理以前难以想象的大量文本的艰巨挑战。集群计算是解决规模问题的唯一实际解决方案。该项目利用MapReduce框架（通过开源Hadoop实现）来解决信息检索应用程序在处理大量数据时的健壮性和可伸缩性问题。更一般地说，目标是探索大型分布式计算环境中处理器、磁盘、内存和网络之间的关系，在这种环境中，在单个机器中做出的许多假设不再成立。一个相关的例子是Hadoop与实时交互应用程序需求之间的根本不匹配。因为它是为面向吞吐量的批处理而设计的，Hadoop目前不提供实时搜索所需的低延迟磁盘访问。分布式内存中对象缓存架构为这个问题提供了一个潜在的解决方案。为了获得更广泛的影响，这项研究的结果将在Ivory中实现，Ivory是一个分布式信息检索的开源工具包，从底层开始构建，考虑到集群架构。该工具包的可用性将有助于维持“云计算”这一新兴领域的活动。更多信息请访问项目网站（http://www.umiacs.umd.edu/~jimmylin/cloud-computing）。