Modeling inter-rater agreement using mixed models

描述（由申请人提供）：在常见诊断医疗程序中对患者的评估，例如通过查看乳房X线照片诊断乳腺癌，通常基于医生的专家意见，其中经验丰富的评估者之间的高度一致性表明诊断程序准确。在这些主观类型的分类中，评级者之间通常会观察到实质性的差异。这种担忧促使研究人员开发统计方法来评估诊断程序的可靠性。目前评估评分员间一致性的方法，包括科恩的Kappa，容易产生偏差，通常将评分员和项目建模为固定效应，因此不允许对一般过程进行推断，并且不容易纳入多个评分员，二分法结果或不平衡数据。拟议的研究的主要重点是准确地衡量评分员之间的协议，在一个灵活和现实的方式，产生一个一般的基本医疗诊断过程的推断，并确定影响评级过程的重要因素。因此，这些信息可用于培训医生和其他生物医学专业人员，以提高他们的诊断技能。将使用所提出的方法分析来自一些研究的数据，这些研究测量合格医生在诊断癌症和其他疾病方面的一致性，包括诊断乳腺癌的乳房X线照片，评估前列腺癌的格里森分级量表。模型可以包含多个评分者和项目，以及二分结果（存在/不存在疾病）。在这些研究中，评价者间的一致性的解释将被强调。拟议的研究的一个重要特点是开发的协议，这是很容易解释的生物医学专业人士，并避免使用科恩的kappa统计中观察到的缺陷的总体措施。将进行广泛的模拟研究，以评估统计方法的性能。将开发并公开提供便于用户使用的软件，以适应拟议的模型和一致性衡量标准。

DESCRIPTION (provided by applicant): Assessment of patients in common diagnostic medical procedures, such as the diagnosis of breast cancer from viewing a mammogram, is often based on the expert opinion of physicians, where strong agreement between experienced raters is suggestive of an accurate diagnostic procedure. Substantial variability is commonly observed between raters in these subjective types of classifications. This concern has prompted researchers to develop statistical methods to assess the reliability of diagnostic procedures. Current methods for assessing inter-rater agreement, including Cohen's kappa, are prone to bias, usually model the raters and items as fixed effects thus not allowing inference about the general process, and do not easily incorporate multiple raters, dichotomous outcomes or unbalanced data. The primary focus of the proposed research is to accurately measure agreement between raters in a flexible and realistic manner, to yield inference about a general underlying medical diagnostic process, and to identify important factors that influence the rating process. This information can consequently be used in the training of physicians and other biomedical professionals to improve their diagnosis skills. Data from a number of studies measuring agreement between qualified physicians in the diagnosis of cancers and other diseases, including mammograms for diagnosing breast cancer, the Gleason grading scale for assessing prostate cancers, will be analyzed using the proposed methodology. The models can incorporate multiple raters and items, and dichotomous outcomes (presence/absence of disease). Interpretation of inter-rater agreement in these studies will be emphasized. An important feature of the proposed research is the development of an overall measure of agreement which is easily interpretable by biomedical professionals and avoids flaws observed in the use of Cohen's kappa statistic. Extensive simulation studies will be carried out to assess the performance of the statistical methods. User-friendly software to fit the proposed models and measure of agreement will be developed and made publicly available.