Searching Documents with Text and Mathematical Content Using a Pen-Based Interface

科学和工程文档通常既包含文本又包含数学。然而，在搜索这类文档时，当前的商业技术只能匹配文本，因此不太适合查找数学表达式等任务；必须扩展这一功能以使其具有数学意识。在本方案中，我们的目标是同时搜索文本和数学内容，以获得最佳搜索结果。 已经表明，当搜索引擎能够为文档中出现的术语推断某些语义时，它们变得更有效，因为在这种情况下，具有相似含义的术语可以被匹配。因此，我们希望确保数学感知搜索能够推断出数学公式的语义。通过这种方式，基于文本和数学公式的检索可以变得更有效。 为了让一个有数学意识的搜索引擎产生广泛的影响，重要的是用户能够以自然的手写方式输入他们的数学公式，而不是使用一维LaTeX之类的命令。这在当今基于笔的设备上是很自然的，因此我们项目的一个重要部分是研究如何使用基于笔的界面来检索包含丰富数学内容的文档。 将手写数学表达式转换为在数学感知搜索系统中使用是相当困难的。挑战包括识别单个字符、将手写字符组合成数学表达式、区分相似的表情、使用手势和确定通配符。我们计划研究数据驱动的方法，以确保对手写数学的良好识别。 我们的数学意识检索系统的实施和评估在我们的工作中起着核心作用。

Scientific and engineering documents typically contain both text and mathematics. However, when it comes to searching such documents, current commercial technology is able to match text only and so is not well-suited for tasks such as finding mathematical expressions; this must be extended to become math-aware. In this proposal we aim to search text and mathematical content together in order to get the best search results. It has been shown that search engines become more effective when they are able to infer some semantics for terms appearing in documents since, in such instances, terms with similar meanings can be matched. As a result we wish to ensure that math-aware search can infer semantics for mathematical formulas. In this way retrieval based on both text and mathematical formulas can become more effective. In order for a math-aware search engine to have broad impact, it is important that a user be able to enter their mathematical formulas in a natural handwritten manner rather than to use one-dimensional LaTeX-like commands. This can be natural on today's pen-based devices and so an important part of our project is to investigate how a pen-based interface can be used to retrieve documents that are rich in mathematical content. Converting handwritten mathematical expressions to be used in a math-aware search system has considerable difficulties. Challenges include recognition of individual characters, combining handwritten characters into math expressions, distinguishing among similar looking expressions, use of gestures, and determining wild cards. We plan to investigate data-driven methods to ensure good recognition of handwritten math. The implementation and evaluation of our math-aware retrieval system plays a central role in our work.