Overview of the ID, EPI and REL tasks of BioNLP Shared Task 2011.
Overview of the ID, EPI and REL tasks of BioNLP Shared Task 2011.
复制标题
DOI：
10.1186/1471-2105-13-s11-s2
复制
发表时间：
2012-06-26
影响因子：
3
通讯作者：
Ananiadou S
中科院分区：
文献类型：
--
作者：
Pyysalo S;Ohta T;Rak R;Sullivan D;Mao C;Wang C;Sobral B;Tsujii J;Ananiadou S
我们介绍了BioNLP共享任务2011的三个任务的准备，资源，结果和分析：传染病（ID）和表观遗传学和翻译后修饰（EPI）的主要任务，以及实体关系（REL）的支持任务。这两个主要任务代表了BioNLP共享任务2009（ST'09）中引入的事件提取模型扩展到生物医学科学文献的两个新领域，每个领域都是由特定生物定位任务的需求驱动的。ID任务涉及感染、毒力和抗性的分子机制，特别关注细菌中普遍存在的一类信号系统的功能。EPI任务致力于提取关于DNA和蛋白质化学修饰的陈述，特别强调与基因表达的表观遗传控制有关的变化。与这两个面向应用的主要任务相比，REL任务旨在通过将与部分关系相关的挑战分离为可以由独立系统解决的子问题来支持提取。七个小组分别参与两项主要任务，四个小组参与辅助任务。参与系统表明了事件提取方法能力的进步，并在许多方面表现出概括性：从摘要到全文，从之前考虑的子域到新子域，以及从ST'09提取目标到其他实体和事件。在支持任务REL，58%的F-分数，实现了最高的性能，与其他关系提取任务报告的水平大致相当。对于ID任务，性能最高的系统获得了56%的F分数，与已建立的ST'09任务的最先进性能相当。在EPI任务中，最好的结果是对于完整的提取目标集的53%的F分数和对于减少的核心提取目标集的69%的F分数，接近足以用于面向用户的应用的性能水平。在这项研究中，我们扩展了先前报告的结果，并对参与系统的输出进行了进一步的分析。我们特别强调与现实世界的适用性有关的系统性能方面，考虑替代评估指标，并对系统输出进行额外的手动分析。我们进一步证明，提取系统的优势可以结合起来，以提高任何孤立的系统所实现的性能。所有任务的手动注释语料库、支持资源和评估工具均可从http://www.bionlp-st.org获得，这些任务将继续作为所有相关方的开放挑战。
We present the preparation, resources, results and analysis of three tasks of the BioNLP Shared Task 2011: the main tasks on Infectious Diseases (ID) and Epigenetics and Post-translational Modifications (EPI), and the supporting task on Entity Relations (REL). The two main tasks represent extensions of the event extraction model introduced in the BioNLP Shared Task 2009 (ST'09) to two new areas of biomedical scientific literature, each motivated by the needs of specific biocuration tasks. The ID task concerns the molecular mechanisms of infection, virulence and resistance, focusing in particular on the functions of a class of signaling systems that are ubiquitous in bacteria. The EPI task is dedicated to the extraction of statements regarding chemical modifications of DNA and proteins, with particular emphasis on changes relating to the epigenetic control of gene expression. By contrast to these two application-oriented main tasks, the REL task seeks to support extraction in general by separating challenges relating to part-of relations into a subproblem that can be addressed by independent systems. Seven groups participated in each of the two main tasks and four groups in the supporting task. The participating systems indicated advances in the capability of event extraction methods and demonstrated generalization in many aspects: from abstracts to full texts, from previously considered subdomains to new ones, and from the ST'09 extraction targets to other entities and events. The highest performance achieved in the supporting task REL, 58% F-score, is broadly comparable with levels reported for other relation extraction tasks. For the ID task, the highest-performing system achieved 56% F-score, comparable to the state-of-the-art performance at the established ST'09 task. In the EPI task, the best result was 53% F-score for the full set of extraction targets and 69% F-score for a reduced set of core extraction targets, approaching a level of performance sufficient for user-facing applications. In this study, we extend on previously reported results and perform further analyses of the outputs of the participating systems. We place specific emphasis on aspects of system performance relating to real-world applicability, considering alternate evaluation metrics and performing additional manual analysis of system outputs. We further demonstrate that the strengths of extraction systems can be combined to improve on the performance achieved by any system in isolation. The manually annotated corpora, supporting resources, and evaluation tools for all tasks are available from http://www.bionlp-st.org and the tasks continue as open challenges for all interested parties.