An open-source framework for large-scale, flexible evaluation of biomedical text mining systems.
An open-source framework for large-scale, flexible evaluation of biomedical text mining systems.
复制标题
生物医学文本挖掘系统的大规模，灵活评估的开源框架。
DOI：
10.1186/1747-5333-3-1
复制
发表时间：
2008-01-29
期刊：
影响因子：
--
通讯作者：
Hunter L
中科院分区：
文献类型：
--
作者：
Baumgartner WA Jr;Cohen KB;Hunter L
改进评价方法是提高文本挖掘理论和实践水平的必要前提。本文提出了一个公开可用的框架，它有助于对文本挖掘技术进行全面、结构化和大规模的评估。该框架的可扩展性及其通过分析组件部件揭示系统范围特征的能力，以及它在促进第三方应用程序集成方面的有用性，都通过生物医学领域的示例得到了证明。我们的评估框架是使用非结构化信息管理体系结构组装的。运用该方法对225个系统、评价语料库和正确性测度组合的基因提及识别系统进行了分析。研究发现，这三者之间的相互作用会影响系统的相对排名。第二个实验评估了基因规范化系统的性能，使用4097个基因提及系统和基因提及系统组合策略的组合作为输入。研究表明，基因提及系统召回对基因规范化系统性能的影响远大于基因提及系统精度的影响，而高的基因规范化性能可以在极低的基因提及系统精度水平下实现。本文中介绍的软件展示了生物医学语言处理系统的结构化评估所产生的新发现的潜力，以及这种评估框架对促进生物医学语言处理技术开发人员之间协作的有用性。代码库是SourceForge.net上BioNLP UIMA组件存储库的一部分。
Improved evaluation methodologies have been identified as a necessary prerequisite to the improvement of text mining theory and practice. This paper presents a publicly available framework that facilitates thorough, structured, and large-scale evaluations of text mining technologies. The extensibility of this framework and its ability to uncover system-wide characteristics by analyzing component parts as well as its usefulness for facilitating third-party application integration are demonstrated through examples in the biomedical domain. Our evaluation framework was assembled using the Unstructured Information Management Architecture. It was used to analyze a set of gene mention identification systems involving 225 combinations of system, evaluation corpus, and correctness measure. Interactions between all three were found to affect the relative rankings of the systems. A second experiment evaluated gene normalization system performance using as input 4,097 combinations of gene mention systems and gene mention system-combining strategies. Gene mention system recall is shown to affect gene normalization system performance much more than does gene mention system precision, and high gene normalization performance is shown to be achievable with remarkably low levels of gene mention system precision. The software presented in this paper demonstrates the potential for novel discovery resulting from the structured evaluation of biomedical language processing systems, as well as the usefulness of such an evaluation framework for promoting collaboration between developers of biomedical language processing technologies. The code base is available as part of the BioNLP UIMA Component Repository on SourceForge.net.