III: Small: Combining Algorithms for Recognition and Retrieval of Mathematics

目前在美国有一个显著的需要增加数学素养。除其他好处外，这将允许更多的美国人从事STEM（科学，技术，工程和数学）学科的职业。 在互联网上寻找数学入门信息的最广泛的资源，主要支持基于文本的搜索。 类似地，当前的可移植文档格式（. pdf）查看器支持文本搜索，但不支持math.This研究的目标是开发一个查询的表达式机制，用户使用图像，手写笔/手指，鼠标和键盘输入表达式。然后，用户可以使用表达式外观、符号、结构和数学语义的组合进行搜索。 表达式属性可以与基于文本的搜索相结合，使得对数学信息的查询比当前基于文本的方法更精确。为了使按表达式查询的方法可行，需要改进数学识别，沿着开发用于索引和检索数学表达式的有效方法。特别是，该项目旨在改善手写和打字数学的光学字符识别（OCR），沿着解析表达式结构的方法。 采用基于图Transformer Networks（GTN）和Boosting技术的方法，将数学符号定位、识别和关联模块智能组合，进一步提高数学符号的识别能力。 研究的重点是确定适当的功能，距离度量，索引和搜索方法的表达检索。开发的技术通过用户研究进行评估，包括实验室设置和互联网。还计划进行更多的用户研究，以确定按表达式查询的适当用例。该项目预计将产生数学专家和（也许更重要的是）非专家都可用的新的按表达式查询方法。这些方法可能适用于检索其他非文本文档元素，如化学图表，表格和图形。为该项目开发的源代码和实验数据将通过项目网站（http：//www.cs.rit.edu/msearch.html）公布。为了促进数学素养，该项目的首席研究员和研究生将访问中学，并谈论数学符号的历史，识别和检索。PI还计划参加RIT的McNair学者计划，该计划旨在为有兴趣攻读博士学位的低收入第一代大学生提供研究经验。

Currently there is a significant need for increasing mathematical literacy in the United States. Among other benefits, this would permit more Americans to pursue careers in STEM (Science, Technology, Engineering and Mathematics) disciplines. The most extensive resource for finding introductory information on math, the internet, primarily supports text-based search. Similarly, current Portable Document Format (.pdf) viewers support search for text, but not math. The goal of this research is to develop a query-by-expression mechanism, where users enter expressions using images, stylus/finger, mouse and keyboard. Users may then search using a combination of the expression appearance, symbols, structure, and mathematical semantics. Expression properties may be combined with text-based search, allowing queries for mathematical information to be more precise than current text-based methods.For query-by-expression methods to be viable, improvements in math recognition are needed, along with the development of efficient methods for indexing and retrieving mathematical expressions. In particular, the project seeks to improve optical character recognition (OCR) for handwritten and typeset mathematics, along with methods for parsing expression structure. Approach based on Graph Transformer Networks (GTN) and adaptations of boosting techniques is applied to intelligent combination of modules that locate, recognize and relate mathematical symbols with an aim to further improve recognition. Research focuses on identifying appropriate features, distance metrics, indexing and search methods for expression retrieval. Developed techniques are evaluated via user studies both in-lab settings and through the internet. Additional user studies to identify appropriate use cases for query-by-expression are also planned.This project is expected to produce new query-by-expression methods usable by both math experts and (perhaps more importantly) non-experts. These methods might be adapted to retrieving other non-textual document elements such as chemical diagrams, tables, and figures. Source code and experimental data developed for the project will be made public via the project web site (http://www.cs.rit.edu/~dprl/msearch.html). To promote mathematical literacy, the principal investigator and graduate students working on the project will visit middle schools and talk about the history, recognition and retrieval of mathematical notation. The PI also plans to participate in the McNair Scholars program at RIT, which seeks to provide research experiences to low-income, first-generation college students that are interested in pursuing doctoral studies.