Application of MCAT questions as a testing tool and evaluation metric for knowledge graph-based reasoning systems.
Application of MCAT questions as a testing tool and evaluation metric for knowledge graph-based reasoning systems.
复制标题
MCAT问题作为知识图推理系统的测试工具和评价指标的应用。
DOI：
10.1111/cts.13021
复制
发表时间：
2021-09
期刊：
影响因子：
--
通讯作者：
Ahalt SC
中科院分区：
文献类型：
--
作者：
Fecho K;Balhoff J;Bizon C;Byrd WE;Hang S;Koslicki D;Rensi SE;Schmitt PL;Wawer MJ;Williams M;Ahalt SC
“知识图”（KGs）已成为表示生物医学知识的常用方法。在KG中，多个生物医学数据集可以链接在一起作为图形表示，其中节点表示实体，例如“化学物质”或“基因”，而边表示谓词，例如“原因”或“治疗”。推理和推理算法可以应用于KG并用于生成新的知识。我们开发了三个基于KG的问答系统，作为生物医学数据翻译程序的一部分。这些系统通常使用传统的软件工程工具和方法进行测试和评估。在这项研究中，我们探索了一种基于团队的方法来测试和评估原型“翻译推理”通过应用医学院校入学考试（MCAT）的问题。具体来说，我们描述了三个“黑客马拉松”，其中每个系统的开发人员与主持人一起工作，以确定是否可以使用的应用程序来解决MCAT的问题。结果表明，系统性能逐步提高，在第一次黑客攻击期间正确答案为0%（0/5），在第二次黑客攻击期间正确答案为75%（3/4），在最终黑客攻击期间正确答案为100%（5/5）。我们讨论了技术和社会学的经验教训，并得出结论，MCAT问题可以成功地应用于主持黑客松的背景下，测试和评估原型KG-基于问答系统，确定当前能力的差距，并提高性能。最后，我们强调了翻译者推理者的几个已发表的临床和翻译科学应用。
“Knowledge graphs” (KGs) have become a common approach for representing biomedical knowledge. In a KG, multiple biomedical data sets can be linked together as a graph representation, with nodes representing entities, such as “chemical substance” or “genes,” and edges representing predicates, such as “causes” or “treats.” Reasoning and inference algorithms can then be applied to the KG and used to generate new knowledge. We developed three KG‐based question‐answering systems as part of the Biomedical Data Translator program. These systems are typically tested and evaluated using traditional software engineering tools and approaches. In this study, we explored a team‐based approach to test and evaluate the prototype “Translator Reasoners” through the application of Medical College Admission Test (MCAT) questions. Specifically, we describe three “hackathons,” in which the developers of each of the three systems worked together with a moderator to determine whether the applications could be used to solve MCAT questions. The results demonstrate progressive improvement in system performance, with 0% (0/5) correct answers during the first hackathon, 75% (3/4) correct during the second hackathon, and 100% (5/5) correct during the final hackathon. We discuss the technical and sociologic lessons learned and conclude that MCAT questions can be applied successfully in the context of moderated hackathons to test and evaluate prototype KG‐based question‐answering systems, identify gaps in current capabilities, and improve performance. Finally, we highlight several published clinical and translational science applications of the Translator Reasoners.
登录
查看更多内容
影响因子：
5.6
作者：
Bizon, Chris;Cox, Steven;Tropsha, Alexander
通讯作者：
Tropsha, Alexander
影响因子：
14.9
作者：
Shefchek, Kent A.;Harris, Nomi L.;Osumi-Sutherland, David
通讯作者：
Osumi-Sutherland, David
DOI：
10.1111/cts.12592
发表时间：
2019-03
期刊：
Clinical and translational science
影响因子：
--
作者：
Biomedical Data Translator Consortium
通讯作者：
Biomedical Data Translator Consortium
影响因子：
4.5
作者：
Fecho, Karamarie;Ahalt, Stanley C.;Peden, David B.
通讯作者：
Peden, David B.
影响因子：
5.8
作者：
Morton, Kenneth;Wang, Patrick;Tropsha, Alexander
通讯作者：
Tropsha, Alexander