Evaluation of software impact designed for biomedical research: Are we measuring what’s meaningful?-猫眼课题宝

DOI：

10.48550/arxiv.2306.03255

发表时间：

2023-06

期刊：

ArXiv

影响因子：

通讯作者：

中科院分区：

其他

文献类型：

作者：

Awan Afiaz;Andrey Ivanov;J. Chamberlin;D. Hanauer;Candace Savonen;M. Goldman;M. Morgan;Michael Reich;Alexander Getka;Aaron N. Holmes;Sarthak Pati;D. Knight;P. Boutros;S. Bakas;J. Caporaso;G. Fiol;H. Hochheiser;Brian Haas;P. Schloss;James A. Eddy;Jake Albrecht;A. Fedorov;L. Waldron;Ava M. Hoffman;R. Bradshaw;J. Leek;Carrie Wright Department of Biostatistics;U. Washington;Seattle;Wa;Biostatistics Program;Public Health Sciences Division;Fred Hutchinson Cancer Research Center;Departmentof Pharmacology;Chemical Biology;Emory University School of Medicine;Emory University;Atlanta;Ga;Department of Biomedical Informatics;U. Utah;Salt lake City;Ut;Department of MathematicalStatistical Sciences;University of Michigan Medical School;A. Arbor;Mi;University of California at Santa Cruz;Santa Cruz;Ca;Roswell Park Comprehensive Cancer Center;Buffalo;Ny;U. California;San Diego;La Jolla;U. Pennsylvania;Philadelphia;Pa;Jonsson Comprehensive Cancer Center;Los Angeles;I. Health;Department of Genetics;Department of Urology;Pathogen;M. Institute;Northern Arizona University;Flagstaff;Az;U. Pittsburgh;Pittsburgh;Methods Development Laboratory;Broad Institute;Cambridge;Ma;D. Microbiology;Immunology;U. Michigan;Sage Bionetworks;D. Radiology;Brigham;Women's Hospital;H. School;Boston;D. Epidemiology;Biostatistics;City Health;Health Policy;New York.

软件对生物学和医学的发展至关重要。通过分析软件的使用情况和影响指标，开发人员可以帮助确定用户和社区的参与度。这些指标可以用来证明额外的资金，鼓励额外的使用，并确定未预料到的用例。这样的分析可以帮助定义改进领域，并协助管理项目资源。然而，与评估使用和影响相关的挑战是存在的，其中许多挑战根据所评估的软件类型而变化很大。这些挑战包括扭曲、夸大、低估或误导指标的问题，以及道德和安全问题。需要更多地关注细微差别、挑战，以及在捕获跨生物软件的不同范围的影响时所涉及的考虑。此外，有些工具可能对少数受众特别有益，但可能没有相对引人注目的高使用率指标。尽管一些原则是普遍适用的，但是并没有一个完美的度量或方法来有效地评估软件工具的影响，因为这取决于每个工具的独特方面，它是如何使用的，以及人们希望如何评估参与。我们提出了更广泛适用的指导方针（例如支持软件使用的基础设施和关于使用的度量集合），以及针对各种类型的软件和资源的策略。我们还强调了社区如何度量或评估软件影响方面的突出问题。为了更深入地了解阻碍软件评估的问题，以及确定哪些似乎是有帮助的，我们对参与由国家癌症研究所（NCI）资助的癌症研究信息技术（ITCR）计划的科学软件项目的参与者进行了调查。我们还调查了这个科学社区和其他社区中的软件，以评估支持这种评估的基础设施实现的频率，以及这如何影响描述软件使用情况的论文的比率。我们发现，尽管开发人员认识到分析与软件的影响或使用相关的数据的效用，但他们很难找到时间或资金来支持这种分析。我们还发现，诸如社交媒体的存在、更深入的文档、软件健康指标的存在以及如何联系开发人员的明确信息等基础设施似乎与使用率的增加有关。我们的发现可以帮助科学的软件开发人员充分利用他们的软件评估，这样他们就可以更充分地从这样的评估中获益。

Software is vital for the advancement of biology and medicine. Through analysis of usage and impact metrics of software, developers can help determine user and community engagement. These metrics can be used to justify additional funding, encourage additional use, and identify unanticipated use cases. Such analyses can help define improvement areas and assist with managing project resources. However, there are challenges associated with assessing usage and impact, many of which vary widely depending on the type of software being evaluated. These challenges involve issues of distorted, exaggerated, understated, or misleading metrics, as well as ethical and security concerns. More attention to the nuances, challenges, and considerations involved in capturing impact across the diverse spectrum of biological software is needed. Furthermore, some tools may be especially beneficial to a small audience, yet may not have comparatively compelling metrics of high usage. Although some principles are generally applicable, there is not a single perfect metric or approach to effectively evaluate a software tool’s impact, as this depends on aspects unique to each tool, how it is used, and how one wishes to evaluate engagement. We propose more broadly applicable guidelines (such as infrastructure that supports the usage of software and the collection of metrics about usage), as well as strategies for various types of software and resources. We also highlight outstanding issues in the field regarding how communities measure or evaluate software impact. To gain a deeper understanding of the issues hindering software evaluations, as well as to determine what appears to be helpful, we performed a survey of participants involved with scientific software projects for the Informatics Technology for Cancer Research (ITCR) program funded by the National Cancer Institute (NCI). We also investigated software among this scientific community and others to assess how often infrastructure that supports such evaluations is implemented and how this impacts rates of papers describing usage of the software. We find that although developers recognize the utility of analyzing data related to the impact or usage of their software, they struggle to find the time or funding to support such analyses. We also find that infrastructure such as social media presence, more in-depth documentation, the presence of software health metrics, and clear information on how to contact developers seem to be associated with increased usage rates. Our findings can help scientific software developers make the most out of the evaluations of their software so that they can more fully benefit from such assessments.

Evaluation of software impact designed for biomedical research: Are we measuring what’s meaningful?

文献摘要

相似文献