A Distributed System Improves Inter-Observer and AI Concordance in Annotating Interstitial Fibrosis and Tubular Atrophy.
A Distributed System Improves Inter-Observer and AI Concordance in Annotating Interstitial Fibrosis and Tubular Atrophy.
复制标题
分布式系统提高了观察者间和人工智能在注释间质纤维化和肾小管萎缩方面的一致性。
DOI：
10.1117/12.2581789
复制
发表时间：
2021
期刊：
影响因子：
--
通讯作者：
Tomaszewski
中科院分区：
文献类型：
--
作者：
Shashiprakash,AvinashKammardi;Lutnick,Brendon;Ginley,Brandon;Govind,Darshana;Lucarelli,Nicholas;Jen,Kuang-Yu;Rosenberg,AviZ;Urisman,Anatoly;Walavalkar,Vighnesh;Zuckerman,JonathanE;Delsante,Marco;Bissonnette,MeiLinZ;Tomaszewski
间质纤维化和肾小管萎缩（IFTA）的组织学检查对于确定肾脏疾病中不可逆肾损伤的程度至关重要。目前的临床标准涉及病理学家对IFTA的视觉评估，这容易出现观察者间的差异。为了解决这一诊断变异性，我们设计了两个案例研究（CS），包括7名病理学家，使用HistomicsTK-一个分布式系统开发的Kitware公司。（纽约州克利夫顿公园）。将25个全载玻片图像（WSI）分类为21个训练集和4个验证集。训练集由七个独特的子集组成，每个子集与来自验证集的四个常见WSI一起沿着提供给个体病理学家。在CS 1中，所有病理学家在各自的切片中单独注释IFTA。然后，这些注释用于训练深度学习算法，以计算分割IFTA。在CS 2中，注释者首先审查了CS 1的手动和计算注释，以提高IFTA注释的一致性。然后如CS 1中那样重复手动和计算注释过程。通过Krippendorff α（KA）测量验证集中的观察者间一致性。CS 1中7名病理学家的KA为0.62，CI [0.57，0.67]，在CS2中审查彼此的注释后，KA为0.66，CI [0.60，0.72]。当将深度学习者作为第八个注释者时，CS 1和CS 2 KA分别为0.58（CI [0.52，0.64]）和0.63（CI [0.56，0.69]）。这些结果表明，我们设计的注释框架改进了IFTA空间注释的一致性，并展示了一种人类-AI方法，可以显着改善计算模型的开发。
Histologic examination of interstitial fibrosis and tubular atrophy (IFTA) is critical to determine the extent of irreversible kidney injury in renal disease. The current clinical standard involves pathologist’s visual assessment of IFTA, which is prone to inter-observer variability. To address this diagnostic variability, we designed two case studies (CSs), including seven pathologists, using HistomicsTK- a distributed system developed by Kitware Inc. (Clifton Park, NY). Twenty-five whole slide images (WSIs) were classified into a training set of 21 and a validation set of four. The training set was composed of seven unique subsets, each provided to an individual pathologist along with four common WSIs from the validation set. In CS 1, all pathologists individually annotated IFTA in their respective slides. These annotations were then used to train a deep learning algorithm to computationally segment IFTA. In CS 2, manual and computational annotations from CS 1 were first reviewed by the annotators to improve concordance of IFTA annotation. Both the manual and computational annotation processes were then repeated as in CS1. The inter-observer concordance in the validation set was measured by Krippendorff’s alpha (KA). The KA for the seven pathologists in CS1 was 0.62 with CI [0.57, 0.67], and after reviewing each other’s annotations in CS2, 0.66 with CI [0.60, 0.72]. The respective CS1 and CS2 KA were 0.58 with CI [0.52, 0.64] and 0.63 with CI [0.56, 0.69] when including the deep learner as an eighth annotator. These results suggest that our designed annotation framework refines agreement of spatial annotation of IFTA and demonstrates a human-AI approach to significantly improve the development of computational models.