Scalable and Robust GRID-based Text Mining of Scientific Papers

搜索引擎技术是最赚钱的信息技术行业之一。目前的搜索技术是基于关键字的使用，当应用于复杂的信息源时具有严重的限制。先进的文本挖掘技术从文档中提取更多有用的信息，并有望极大地提高用户搜索的成功率。我们计划部署这种技术来搜索科学文献，作为在其他领域进行更大规模商业部署的先导。这需要扩大现有工具的容量，使使用计算网格成为一种有吸引力的可能性。分布式技术很容易应用，因为每个文档都可以以独立的方式进行处理。然而，早期的实验表明，文本处理流水线的健壮性和分布式作业管理存在许多问题，导致无法接受的大量论文未被批注。该项目的主要目标是通过使用STFC资助的恒河系统开发更好的基于网格的作业管理工具，以及通过重新实施文本挖掘系统中可靠性较低的组件，以更强大和更通用的方式解决这些问题。由此产生的系统应该允许合作伙伴公司为目前市场上的任何公司提供优越的搜索工具，并可能带来非常大的商业回报。

Search engine technologies are one of the most profitable information technology sectors. Current search techniques are based on the use of keywords and have serious limitations when applied to complex information sources. Advanced text-mining techniques extract far more useful information from documents, and promise to vastly improve the success of users' searches. We plan to deploy such techniques to search scientific literature as a precursor to a larger commercial deployment in other domains. This requires the existing tools to be scaled up in capacity, making the use of computational grids an attractive possibility. Distributed techniques are easily applicable as each document can be processed in a self-contained way. However, early experiments have revealed a number of problems with the robustness of the text processing pipeline and with distributed job management which result in an unacceptably high number of papers not being annotated. The main goal of this project is to address these issues in a more robust and generic way by developing better Grid-based job management tools using the STFC-funded Ganga system, as well as by re-implementing the less reliable components of the text mining system. The resulting system should allow the partner company to offer a superior search facility to any on the market at the moment, with potentially very large commercial returns.