Learning Word Relationships Using TupleFlow

智能信息检索中心（CIIR）正在研究统计得出的语义词关系对信息检索的影响。利用这些关系，例如，通过识别不同的词何时表达相同的内容，可以导致更有效的检索结果排名。语义关系在文本中没有明确的标记，并且变化太大，无法仅用手识别。CIIR正在使用离线和检索时计算来挖掘大量语料库中的直接和间接词共现数据。特别关注的是技术，创建和使用基于Web的语料库的“可比”的句子和文本块估计单词和短语的翻译概率，并从“上下文向量”，代表单词和短语的含义的关系的技术。正在使用大规模检索实验来测试所发现的单词关系的质量。此外，CIIR正在通过将其新的分布式计算框架TupleFlow迁移到Hadoop来解决大规模数据挖掘的计算障碍。该框架是为文本中关系结构的大规模研究所需的索引和分析操作类型而开发的。TupleFlow是MapReduce的扩展，具有灵活性，可伸缩性，磁盘抽象和低抽象损失方面的优势。预计这项工作将通过提高搜索结果的质量产生广泛影响。

The Center for Intelligent Information Retrieval (CIIR) is investigating the impact of statistically derived semantic word relationships on information retrieval. Exploiting these relationships, for example, by identifying when different words express the same content can lead to more effective rankings of retrieval results. Semantic relationships are not labeled explicitly in text and are too varied to be identified solely by hand. The CIIR is mining massive corpora for direct and indirect word co-occurrence data using both offline and retrieval-time computation. The particular focus is on techniques that create and use Web-based corpora of "comparable" sentences and text chunks for estimating word and phrase translation probabilities, and on techniques that derive relationships from "context vectors" that represent word and phrase meanings. The quality of the word relationships that are discovered is being tested using large-scale retrieval experiments. In addition, the CIIR is addressing computational barriers to large-scale data mining by moving its new distributed computational framework, TupleFlow, to Hadoop. That framework was developed for the type of indexing and analysis operations that are required for large-scale studies of relational structure in text. TupleFlow is an extension of MapReduce, with advantages in flexibility, scalability, disk abstraction, and low abstraction penalties. This work is expected to have broad impact by improving the quality of search results.