The lung image database consortium (LIDC): Ensuring the integrity of expert-defined "truth"
The lung image database consortium (LIDC): Ensuring the integrity of expert-defined "truth"
复制标题
DOI：
10.1016/j.acra.2007.08.006
复制
发表时间：
2007-12-01
影响因子：
4.8
通讯作者：
Clarke, Laurence P.
中科院分区：
文献类型：
--
作者：
Armato, Samuel G., III;Roberts, Rachael Y.;Clarke, Laurence P.
理由和目标。计算机辅助诊断（CAD）系统从根本上需要专家人类观察员的意见，以建立算法开发，培训和测试的“真理”。然而，在调查人员承诺将这一“黄金标准”作为他们研究的基础之前，必须确定这一“真理”的完整性。本研究的目的是开发一个质量保证（QA）模型作为一个不可分割的组成部分的“真相”收集过程中观察到的肺结节的位置和空间范围的计算机断层扫描（CT）扫描被列入肺图像数据库联盟（LIDC）的公共database.Materials和Methods。由四名放射科医生通过两个阶段的过程解释了100个CT扫描。对于第一个读片阶段（“盲读阶段”），放射科医生独立地识别和注释病变，将每个病变分配到三个类别之一：“结节>= 3 mm”，“结节= 3 mm”。对于第二次读片（“非盲读片阶段”），相同的放射科医生独立评价相同的CT扫描，但提供了来自先前执行的盲读片的所有注释;每个放射科医生可以添加、编辑或删除自己的标记;改变自己标记的病变类别;或保持标记不变。将揭盲后读取的标记集分组为离散结节，并进行QA过程，包括识别完整图像注释过程中引入的潜在错误并纠正这些错误。定义了7类潜在错误;任何标记满足其中一个类别标准的结节都被提交给放射科医生，放射科医生分配该标记以进行校正或确认标记是有意的。在100次CT扫描中的45次（45.0%）中，共发现了105个QA问题。放射科医师审查导致对101个（96.2%）潜在错误进行了修改。21个病灶在揭盲读数后被错误标记为肺结节，通过QA过程删除了该名称。结论。“真相”的建立必须包括QA过程，以保证数据集的完整性，这些数据集将为CAD系统的开发、培训和测试提供基础。
Rationale and Objectives. Computer-aided diagnostic (CAD) systems fundamentally require the opinions of expert human observers to establish "truth" for algorithm development, training, and testing. The integrity of this "truth," however, must be established before investigators commit to this "gold standard" as the basis for their research. The purpose of this study was to develop a quality assurance (QA) model as an integral component of the "truth" collection process concerning the location and spatial extent of lung nodules observed on computed tomography (CT) scans to be included in the Lung Image Database Consortium (LIDC) public database.Materials and Methods. One hundred CT scans were interpreted by four radiologists through a two-phase process. For the first of these reads (the "blinded read phase"), radiologists independently identified and annotated lesions, assigning each to one of three categories: "nodule >= 3 mm," "nodule = 3 mm." For the second read (the "unblinded read phase"), the same radiologists independently evaluated the same CT scans, but with all of the annotations from the previously performed blinded reads presented; each radiologist could add to, edit, or delete their own marks; change the lesion category of their own marks; or leave their marks unchanged. The post-unblinded read set of marks was grouped into discrete nodules and subjected to the QA process, which consisted of identification of potential errors introduced during the complete image annotation process and correction of those errors. Seven categories of potential error were defined; any nodule with a mark that satisfied the criterion for one of these categories was referred to the radiologist who assigned that mark for either correction or confirmation that the mark was intentional.Results. A total of 105 QA issues were identified across 45 (45.0%) of the 100 CT scans. Radiologist review resulted in modifications to 101 (96.2%) of these potential errors. Twenty-one lesions erroneously marked as lung nodules after the unblinded reads had this designation removed through the QA process.Conclusions. The establishment of "truth" must incorporate a QA process to guarantee the integrity of the datasets that will provide the basis for the development, training, and testing of CAD systems.