Scalable and Robust GRID-based Text Mining of Scientific Papers

搜索引擎技术是最赚钱的信息技术领域之一。当前的搜索技术是基于关键字的使用，当应用于复杂的信息源时有严重的局限性。先进的文本挖掘技术从文档中提取出更多有用的信息，并有望大大提高用户搜索的成功率。我们计划将这种技术用于搜索科学文献，作为在其他领域进行更大规模商业部署的先驱。这就要求现有工具的容量扩大，使计算网格的使用成为一种有吸引力的可能性。分布式技术很容易应用，因为每个文档都可以以自包含的方式进行处理。然而，早期的实验揭示了文本处理管道的鲁棒性和分布式作业管理的一些问题，这些问题导致大量论文没有被注释，这是不可接受的。该项目的主要目标是通过使用stfc资助的Ganga系统开发更好的基于网格的作业管理工具，以及通过重新实现文本挖掘系统中不太可靠的组件，以更强大和通用的方式解决这些问题。由此产生的系统应该允许合作公司提供比目前市场上任何公司都优越的搜索设施，并具有潜在的巨大商业回报。

Search engine technologies are one of the most profitable information technology sectors. Current search techniques are based on the use of keywords and have serious limitations when applied to complex information sources. Advanced text-mining techniques extract far more useful information from documents, and promise to vastly improve the success of users' searches. We plan to deploy such techniques to search scientific literature as a precursor to a larger commercial deployment in other domains. This requires the existing tools to be scaled up in capacity, making the use of computational grids an attractive possibility. Distributed techniques are easily applicable as each document can be processed in a self-contained way. However, early experiments have revealed a number of problems with the robustness of the text processing pipeline and with distributed job management which result in an unacceptably high number of papers not being annotated. The main goal of this project is to address these issues in a more robust and generic way by developing better Grid-based job management tools using the STFC-funded Ganga system, as well as by re-implementing the less reliable components of the text mining system. The resulting system should allow the partner company to offer a superior search facility to any on the market at the moment, with potentially very large commercial returns.