Modeling inter-rater agreement using mixed models

使用混合模型对评估者间的一致性进行建模

• 批准号: 7091213
• 负责人: KERRIE P NELSON
• 金额: $ 7.2万
• 依托单位: UNIVERSITY OF SOUTH CAROLINA AT COLUMBIA
• 依托单位国家: 美国
• 财政年份: 2006
• 资助国家: 美国
• 起止时间: 2006-08-21 至 2008-07-31
• 项目状态: 已结题
• 来源: https://reporter.nih.gov/project-details/7091213
• 关键词: classification; model; physicians

DESCRIPTION (provided by applicant): Assessment of patients in common diagnostic medical procedures, such as the diagnosis of breast cancer from viewing a mammogram, is often based on the expert opinion of physicians, where strong agreement between experienced raters is suggestive of an accurate diagnostic procedure. Substantial variability is commonly observed between raters in these subjective types of classifications. This concern has prompted researchers to develop statistical methods to assess the reliability of diagnostic procedures. Current methods for assessing inter-rater agreement, including Cohen's kappa, are prone to bias, usually model the raters and items as fixed effects thus not allowing inference about the general process, and do not easily incorporate multiple raters, dichotomous outcomes or unbalanced data. The primary focus of the proposed research is to accurately measure agreement between raters in a flexible and realistic manner, to yield inference about a general underlying medical diagnostic process, and to identify important factors that influence the rating process. This information can consequently be used in the training of physicians and other biomedical professionals to improve their diagnosis skills. Data from a number of studies measuring agreement between qualified physicians in the diagnosis of cancers and other diseases, including mammograms for diagnosing breast cancer, the Gleason grading scale for assessing prostate cancers, will be analyzed using the proposed methodology. The models can incorporate multiple raters and items, and dichotomous outcomes (presence/absence of disease). Interpretation of inter-rater agreement in these studies will be emphasized. An important feature of the proposed research is the development of an overall measure of agreement which is easily interpretable by biomedical professionals and avoids flaws observed in the use of Cohen's kappa statistic. Extensive simulation studies will be carried out to assess the performance of the statistical methods. User-friendly software to fit the proposed models and measure of agreement will be developed and made publicly available.

描述(由申请人提供)：对常见诊断医疗程序中的患者进行评估，例如通过查看乳房X光照片诊断乳腺癌，通常基于医生的专家意见，经验丰富的评分员之间的强烈一致表明有一种准确的诊断程序。在这些主观类型的分类中，评分者之间通常会观察到很大的差异。这种担忧促使研究人员开发统计方法来评估诊断程序的可靠性。目前用于评估评分者之间一致性的方法，包括科恩的kappa，容易产生偏见，通常将评分者和项目建模为固定的影响，从而不允许对一般过程进行推断，并且不容易合并多个评分者、两种结果或不平衡的数据。拟议研究的主要重点是以灵活和现实的方式准确衡量评分者之间的一致性，得出关于一般潜在医疗诊断过程的推断，并确定影响评分过程的重要因素。因此，这些信息可用于培训医生和其他生物医学专业人员，以提高他们的诊断技能。来自多项研究的数据将使用拟议的方法进行分析，这些研究衡量合格医生在诊断癌症和其他疾病方面的一致性，包括诊断乳腺癌的乳房X光检查，评估前列腺癌的格里森分级标准。这些模型可以包含多个评分者和项目，以及两种结果(有无疾病)。在这些研究中，将强调对评分者之间协议的解释。这项拟议研究的一个重要特征是开发了一种全面的一致性测量方法，该方法很容易被生物医学专业人员解释，并避免了在使用科恩的卡帕统计量时观察到的缺陷。将进行广泛的模拟研究，以评估统计方法的性能。将开发用户友好的软件，以适应拟议的模式和协议措施，并向公众提供。