AF: DC: Collaborative Research: Pattern Matching for Massive Data Sets

模式匹配是一个基础性的研究领域，在生物序列比对、网络搜索引擎和网络入侵检测等领域有着广泛的应用。给定一个模式P和一个文本字符串T，中心问题是在T中找到P的匹配项。当数据变得庞大时，我们不能假设文本可以存储在RAM中。模式匹配问题现在必须用更合适的模型来考虑，如外部内存模型、缓存无关模型、流模型、MapReduce范式和多核模型。在许多情况下，需要开发混合的模型或更新、更合适的模型，以保持应用程序的实际方面。本项目的重点是开发高效的搜索算法和索引，当数据集驻留在磁盘上、网络存储上或只能作为在线流访问时。数据必须是可有效搜索的，即使它可能是压缩格式的。该项目考虑了传统的模式匹配问题，以及各种变体，例如(I)近似匹配--其中模式可能不完全匹配T中的子字符串，(Ii)在线匹配--其中模式(S)是预先已知的，并且文本以流的形式到来，以及(Iii)字符串检索--其中不是查找所有匹配项，而是专注于检索包含查询模式的一个或多个匹配项的高级文档。I/O效率和空间利用率问题是该项目的核心问题。这包括开发合适的海量数据集模型，得出最优的理论界限，并实施实用工具。方法包括模式匹配中的组合和随机方法、简洁的数据结构、top-k查询处理和I/O高效索引。该项目将在模式匹配方面建立新的坚实的理论基础，并直接应用于数据库和信息检索等领域。它将极大地推动网络搜索引擎技术(通过影响倒排索引的使用方式)和基因组序列比对工具(例如BLAST)的当前技术水平。在该项目期间开发的工具和软件将广泛分发给研究界。一些组成部分将作为实施项目纳入本科生和研究生的算法课程课程。

Pattern matching is a fundamental research field with applications in domains such as biological sequence alignment, web search engines and network intrusion detection. Given a pattern P and a text string T, the central problem is to find occurrences of P in T. When data becomes massive, we cannot assume that text can be stored in RAM. Pattern matching problems must now be considered with more appropriate models like external memory model, cache-oblivious model, streaming models, MapReduce paradigm and multi-core models. In many cases, a blend of models or newer, more appropriate models need to be developed keeping the practical aspects of the application in sight.The focus of this project is to develop efficient search algorithms and indexes when a data set resides on disks, on network storage, or is accessible only as an online stream. The data must be efficiently searchable even though it may be in compressed format. The project considers traditional pattern matching problem, as well as variants such as (i) approximate matching -- where the pattern may not exactly match a substring in T, (ii) online matching -- where the pattern(s) are known in advance and text comes as a stream, and (iii) string retrieval -- where instead of finding all the occurrences, the focus is on retrieving high ranking documents which contain one or more occurences of the query pattern. Issues of I/O efficiency and space utilization are central to this project. This involves developing suitable massive data set models, deriving optimal theoretical bounds and implementing practical tools. Methodologies include combinatorial and randomized methods in pattern matching, succinct data structures, top-k query processing and I/O efficient indexes.The project will build new, solid theoretical foundations in pattern matching, with direct applications to fields like databases and information retrieval. It will significantly drive forward current state of the art in web search engine technology (by impacting the way inverted indexes are used) and genome sequence alignment tools (e.g., BLAST). Tools and software developed during this project will be widely distributed to the research community. Some components will be incorporated into undergraduate and graduate algorithms course curricula as implementation projects.