Training Effects on Observer Operating Points in Technology Assessment Studies

描述(申请人提供)：回溯性观察者表现研究已成为对临床实践中预期表现的推论最常用的方法，因为观察者成为诊断系统的组成部分的前瞻性评估是昂贵的、难以执行的、耗时的，并且通常在新技术和/或实践已被大量用户批准和实践一段时间之后进行。人们普遍认为，回溯性研究的结果可以用来推断，至少在相对范围内，应该 发生在实际的临床实践中。不幸的是，先前的研究表明，观察者在回溯性研究中的行为与在解释影响患者护理的实际临床检查时显著不同。与这些推论有关的一个重要问题是观察员的培训。虽然有迹象表明，在实验室中，疾病流行率对成绩的影响很小(如果有的话)，但训练的影响，以及更重要的是，训练组中疾病流行率可能产生的影响尚未得到调查。这可能是最重要的，因为在实施新技术或实践时，不仅是整体性能，而且是“操作点”，对临床应用非常重要。如果一个新的系统或做法确实更好(即，沿着“更高”的曲线执行)，在实际提高敏感性和/或特异性(或两者的组合)之间的权衡可能在很大程度上取决于在研究之前如何对观察员进行专门培训(即，在培训期间特别强调敏感性或特异性)。这项研究要检验的主要假设是，训练组中的TAT疾病患病率可能而且很可能会影响观察者的实际操作点，但观察者将主要沿着由成像系统或实践(固有的)而不是特定的训练组确定的性能曲线(ROC或FROC类型)进行操作。因此，临床实践也可能受到培训期间特别强调的显著影响。我们建议通过进行一项独特的观察者研究来调查这一问题，在该研究中，在阅读相同的病例集时，在不同的训练集中将呈现显著不同的疾病流行水平，同时进行测试。如果我们的主要假设得到证实，那么，通过提供有特定重点的培训，特定的培训可以用于通过强调所需的参数(即，敏感性、特异性或两者的组合)来优化预期的临床实践。因此，拟议的调查对于接受新技术和做法的评估和核准过程中的观察员研究将是极其重要的。一个实际的例子，导致一项新技术的批准推迟了两年以上，因为操作点的“临床上不受欢迎的”转移，直接证明了这项研究的重要性。

DESCRIPTION (provided by applicant): Retrospective observer performance studies have become the most commonly used approach for generating inferences as to the expected performance in the clinical practice, since prospective assessments in which the observer becomes an integral part of the diagnostic system are expensive, difficult to perform, time consuming, and are generally performed after a new technology and / or a practice has been approved and been in practice for a while by a large number of users. The general belief is that findings from retrospective studies can be used to infer, at least on a relative scale, what should occur in actual clinical practice. Unfortunately, previous studies have shown that observers behave significantly different in retrospective studies than during interpretations of actual clinial examinations that affect patient care. One of the important issues related to these inferences is the training of observers. While there are indications that, in the laboratory, disease prevalence has little, if any, effect on performance, the effect of training and, as important, the possible effect of disease prevalence in the training set has not been investigated. This may be of utmost importance, as not only the overall performance but also the "operating point", when implementing new technologies or practices is very important for clinical applications. If a new system or practice is indeed better (i.e., performs along a "higher" curve), the tradeoff between actually increasing sensitivity and / or specificity (or a combination of both) may largely depend on how observers are specifically trained prior to a study (i.e., a particular emphasis on sensitivity or specificity during training). The primary hypothesis to be tested in this study is tat disease prevalence in training sets could, and likely would, affect actual operating points of observers, but observers will largely operate along a performance curve (ROC or FROC type) that is determined by (inherent to) the imaging system or practice rather than the specific training set. Therefore, a clinical practice could also be significantly affected by a specific emphasis during training. We propose to investigate this issue by performing a unique observer study in which substantially different disease prevalence levels will be presented in different training sets while being tested when reading the same case set. If our primary hypothesis is proved, then, by providing training with a specific emphasis, specific training could be used to optimize the intended the clinical practice by emphasizing the desired parameter (i.e., sensitivity, specificity or a combination of both). Thus, the proposed investigation will be extremely important for acceptance of observer studies in the assessment and approval process of new technologies and practices. An actual example that resulted in the delay of approving a new technology by more than two years because of a "clinically undesirable" shift in operating points demonstrates directly the importance of this study.