Learning Word Relationships Using TupleFlow

智能信息检索中心(CIIR)正在调查统计派生的语义词关系对信息检索的影响。利用这些关系，例如，通过识别不同的词何时表达相同的内容可以导致更有效的检索结果排名。语义关系在文本中没有明确的标签，而且变化太大，不能仅用手来识别。CIIR正在使用离线和检索时间计算从海量语料库中挖掘直接和间接单词共现数据。特别关注的是创建和使用基于Web的“可比较”句子和文本块的语料库来估计单词和短语翻译概率的技术，以及从表示单词和短语含义的“上下文向量”推导关系的技术。正在使用大规模的检索实验来测试所发现的单词关系的质量。此外，CIIR正在通过将其新的分布式计算框架TupleFlow迁移到Hadoop来解决大规模数据挖掘的计算障碍。该框架是为对文本中的关系结构进行大规模研究所需的索引和分析操作类型而制定的。TupleFlow是MapReduce的扩展，在灵活性、可伸缩性、磁盘抽象和低抽象惩罚方面具有优势。这项工作有望通过提高搜索结果的质量产生广泛的影响。

The Center for Intelligent Information Retrieval (CIIR) is investigating the impact of statistically derived semantic word relationships on information retrieval. Exploiting these relationships, for example, by identifying when different words express the same content can lead to more effective rankings of retrieval results. Semantic relationships are not labeled explicitly in text and are too varied to be identified solely by hand. The CIIR is mining massive corpora for direct and indirect word co-occurrence data using both offline and retrieval-time computation. The particular focus is on techniques that create and use Web-based corpora of "comparable" sentences and text chunks for estimating word and phrase translation probabilities, and on techniques that derive relationships from "context vectors" that represent word and phrase meanings. The quality of the word relationships that are discovered is being tested using large-scale retrieval experiments. In addition, the CIIR is addressing computational barriers to large-scale data mining by moving its new distributed computational framework, TupleFlow, to Hadoop. That framework was developed for the type of indexing and analysis operations that are required for large-scale studies of relational structure in text. TupleFlow is an extension of MapReduce, with advantages in flexibility, scalability, disk abstraction, and low abstraction penalties. This work is expected to have broad impact by improving the quality of search results.