Constructing and evaluating automated literature review systems
Constructing and evaluating automated literature review systems
复制标题
DOI：
10.1007/s11192-020-03490-w
复制
发表时间：
2020-06
期刊：
影响因子：
3.9
通讯作者：
Jason Portenoy;Jevin D. West
中科院分区：
文献类型：
--
作者：
Jason Portenoy;Jevin D. West
自动化文献综述有可能加速知识综合并提供新的见解。然而，由于缺乏标记的地面实况数据，很难开发和评估这些方法。我们提出了一个框架，该框架使用现有综述论文中的参考文献列表作为标记数据，然后可以用于训练监督分类器，从而可以大规模地对模型和特征进行实验和测试。我们证明了我们的框架，通过训练分类器使用不同的组合，基于引用和文本的功能，对500篇评论论文。我们使用R-Precision分数来重建综述论文的参考文献列表，作为评估和比较方法的一种方式。我们还扩展了我们的方法，生成一组新的文章相关的错误信息研究和科学传播领域。我们发现，我们的方法可以从大量的候选论文中识别出许多最相关的文献综述论文，并且我们的框架允许开发和测试模型和功能，以逐步改善结果。我们构建的模型能够识别相关论文，即使是从一组非常小的种子论文开始。我们还发现，该方法可以适应，以确定以前未发现的文章，可能是相关的一个给定的主题。
Automated literature reviews have the potential to accelerate knowledge synthesis and provide new insights. However, a lack of labeled ground-truth data has made it difficult to develop and evaluate these methods. We propose a framework that uses the reference lists from existing review papers as labeled data, which can then be used to train supervised classifiers, allowing for experimentation and testing of models and features at a large scale. We demonstrate our framework by training classifiers using different combinations of citation- and text-based features on 500 review papers. We use the R-Precision scores for the task of reconstructing the review papers’ reference lists as a way to evaluate and compare methods. We also extend our method, generating a novel set of articles relevant to the fields of misinformation studies and science communication. We find that our method can identify many of the most relevant papers for a literature review from a large set of candidate papers, and that our framework allows for development and testing of models and features to incrementally improve the results. The models we build are able to identify relevant papers even when starting with a very small set of seed papers. We also find that the methods can be adapted to identify previously undiscovered articles that may be relevant to a given topic.