BioCreative III interactive task: an overview.
BioCreative III interactive task: an overview.
复制标题
DOI：
10.1186/1471-2105-12-s8-s4
复制
发表时间：
2011-10-03
影响因子：
3
通讯作者：
Wu CH
中科院分区：
文献类型：
--
作者：
Arighi CN;Roberts PM;Agarwal S;Bhattacharya S;Cesareni G;Chatr-Aryamontri A;Clematide S;Gaudet P;Giglio MG;Harrow I;Huala E;Krallinger M;Leser U;Li D;Liu F;Lu Z;Maltais LJ;Okazaki N;Perfetto L;Rinaldi F;Sætre R;Salgado D;Srinivasan P;Thomas PE;Toldo L;Hirschman L;Wu CH
BioCreative 挑战评估是社区范围内的一项工作，旨在评估应用于生物领域的文本挖掘和信息提取系统。生物策展人社区作为生物医学文献的活跃用户，为文本挖掘工具提供了多元化且积极参与的最终用户群体。早期的 BioCreative 挑战涉及许多文本挖掘团队开发与生物监管相关的基本功能，但他们没有解决系统使用、插入工作流程和监管者采用的问题。因此，在 BioCreative III (BC-III) 中，引入了交互式任务 (IAT)，以解决文本挖掘工具在现实生物管理任务中的实用性和可用性。为了支持 BC-III 中 IAT 的目标，征求了开发人员和最终用户的参与，并要求开发一个用户界面来交互式地处理任务。用户咨询小组（UAG）积极参与IAT设计和评估。该任务的重点是基因标准化（识别文章中提到的基因并将这些基因与标准数据库标识符联系起来）、根据文章中提到的每个基因的总体重要性进行基因排名，以及面向基因的文档检索（识别与所选基因相关的全文论文）。六个系统参与并且都处理和显示同一组文章。这些文章是根据已知对管理有问题的内容进行选择的，例如基因名称的模糊性、多个基因和物种的覆盖范围或引入新的基因名称。 UAG 成员出于培训和评估目的策划了三篇文章，并为每位成员分配了一个系统进行审查。在使用系统来策划文章后，回答了与界面可用性和任务绩效（通过精确度和召回率衡量）相关的调查问卷。尽管分析的文章数量和参与 IAT 实验的用户数量有限，无法对结果进行严格的定量分析，但定性分析为用户在使用系统时遇到的一些问题提供了有价值的见解。总体评估表明，系统可用性功能吸引了大多数用户，但系统性能欠佳（主要是由于基因归一化准确性较低）。其中一些问题包括基因标准化任务中物种鉴定失败和基因名称模糊，导致需要审查大量基因标识符，在某些情况下，这些标识符不包含相关基因。文档检索也存在同样的缺陷。 UAG 赞成实现高性能（通过精确度和召回率来衡量），但强烈建议添加有助于识别正确基因及其标识符的功能，例如帮助消除歧义的上下文信息。 IAT 是一项信息丰富的活动，它促进了策展人和开发人员之间的对话，并增加了对每个小组所面临的挑战的认识。一个主要结论是，目标用户应该积极参与软件开发的每个阶段，这将在未来的任务中得到大力鼓励。 IAT 任务提供了定义指标和功能要求的第一步，这些指标和功能要求是设计 BioCreative IV 挑战中交互式管理系统的正式评估所必需的。
The BioCreative challenge evaluation is a community-wide effort for evaluating text mining and information extraction systems applied to the biological domain. The biocurator community, as an active user of biomedical literature, provides a diverse and engaged end user group for text mining tools. Earlier BioCreative challenges involved many text mining teams in developing basic capabilities relevant to biological curation, but they did not address the issues of system usage, insertion into the workflow and adoption by curators. Thus in BioCreative III (BC-III), the InterActive Task (IAT) was introduced to address the utility and usability of text mining tools for real-life biocuration tasks. To support the aims of the IAT in BC-III, involvement of both developers and end users was solicited, and the development of a user interface to address the tasks interactively was requested. A User Advisory Group (UAG) actively participated in the IAT design and assessment. The task focused on gene normalization (identifying gene mentions in the article and linking these genes to standard database identifiers), gene ranking based on the overall importance of each gene mentioned in the article, and gene-oriented document retrieval (identifying full text papers relevant to a selected gene). Six systems participated and all processed and displayed the same set of articles. The articles were selected based on content known to be problematic for curation, such as ambiguity of gene names, coverage of multiple genes and species, or introduction of a new gene name. Members of the UAG curated three articles for training and assessment purposes, and each member was assigned a system to review. A questionnaire related to the interface usability and task performance (as measured by precision and recall) was answered after systems were used to curate articles. Although the limited number of articles analyzed and users involved in the IAT experiment precluded rigorous quantitative analysis of the results, a qualitative analysis provided valuable insight into some of the problems encountered by users when using the systems. The overall assessment indicates that the system usability features appealed to most users, but the system performance was suboptimal (mainly due to low accuracy in gene normalization). Some of the issues included failure of species identification and gene name ambiguity in the gene normalization task leading to an extensive list of gene identifiers to review, which, in some cases, did not contain the relevant genes. The document retrieval suffered from the same shortfalls. The UAG favored achieving high performance (measured by precision and recall), but strongly recommended the addition of features that facilitate the identification of correct gene and its identifier, such as contextual information to assist in disambiguation. The IAT was an informative exercise that advanced the dialog between curators and developers and increased the appreciation of challenges faced by each group. A major conclusion was that the intended users should be actively involved in every phase of software development, and this will be strongly encouraged in future tasks. The IAT Task provides the first steps toward the definition of metrics and functional requirements that are necessary for designing a formal evaluation of interactive curation systems in the BioCreative IV challenge.