Searching Documents with Text and Mathematical Content Using a Pen-Based Interface

科学和工程文档通常包含文本和数学。然而，在搜索此类文档时，目前的商业技术只能匹配文本，因此不太适合查找数学表达式等任务；这必须扩展到数学意识。在这个建议中，我们的目标是同时搜索文本和数学内容，以获得最佳的搜索结果。研究表明，当搜索引擎能够推断出文档中出现的术语的一些语义时，它们会变得更加有效，因为在这种情况下，具有相似含义的术语可以匹配。因此，我们希望确保数学感知搜索可以推断数学公式的语义。这样，基于文本和数学公式的检索将变得更加有效。为了使数学感知搜索引擎产生广泛的影响，重要的是用户能够以自然的手写方式输入数学公式，而不是使用一维的类似latex的命令。这在今天的基于笔的设备上是很自然的，所以我们项目的一个重要部分是研究如何使用基于笔的接口来检索包含丰富数学内容的文档。将手写的数学表达式转换为在数学感知的搜索系统中使用是相当困难的。挑战包括识别单个字符、将手写字符组合成数学表达式、区分看起来相似的表达式、使用手势以及确定通配符。我们计划研究数据驱动的方法，以确保手写数学的良好识别。数学感知检索系统的实现和评价在我们的工作中起着核心作用。

Scientific and engineering documents typically contain both text and mathematics. However, when it comes to searching such documents, current commercial technology is able to match text only and so is not well-suited for tasks such as finding mathematical expressions; this must be extended to become math-aware. In this proposal we aim to search text and mathematical content together in order to get the best search results. It has been shown that search engines become more effective when they are able to infer some semantics for terms appearing in documents since, in such instances, terms with similar meanings can be matched. As a result we wish to ensure that math-aware search can infer semantics for mathematical formulas. In this way retrieval based on both text and mathematical formulas can become more effective. In order for a math-aware search engine to have broad impact, it is important that a user be able to enter their mathematical formulas in a natural handwritten manner rather than to use one-dimensional LaTeX-like commands. This can be natural on today's pen-based devices and so an important part of our project is to investigate how a pen-based interface can be used to retrieve documents that are rich in mathematical content. Converting handwritten mathematical expressions to be used in a math-aware search system has considerable difficulties. Challenges include recognition of individual characters, combining handwritten characters into math expressions, distinguishing among similar looking expressions, use of gestures, and determining wild cards. We plan to investigate data-driven methods to ensure good recognition of handwritten math. The implementation and evaluation of our math-aware retrieval system plays a central role in our work.