Comparison of Fidelity Assessment Methods

背景资料：国家政策大幅增加了对执行的重视 循证精神卫生服务，满足严重精神疾病患者的需求， VHA在提供有效的、以社区为基础的服务方面取得了长足的进步。其中之一 VHA方法的基石是精神卫生强化病例管理(MHICM)， 该模型基于定义最明确、经验支持最多的方法之一： 果断的社区治疗。最近，VHA政策的转变导致了拟议的一套 提供统一的精神卫生服务，以确保获得一套标准的高质量精神卫生服务 卫生服务，如MHICM，覆盖整个VHA。然而，成功地实施了 广泛的循证实践需要心理测量学上有效而又实用的方法 评估和监测执行程度(即忠诚度)。目前，唯一严格的 监督落实的方法是现场保真走访，这是一项非常耗时的工作， 对于评估者和计划来说，这种方法既昂贵又繁重。 目的：拟议研究的主要目标是检查其有效性 创新和潜在的成本效益方法，以确保心理健康的质量 为患有精神疾病的最严重残疾退伍军人提供服务。具体来说，我们将 检查信度和同时效度、预测效度以及与以下各项相关的成本 三种不同的保真度评估方法：自我报告、基于电话的远程评估、 以及“黄金标准”的现场保真度评估。 方法：目前，所有MHICM团队(n=111)每年都会提供自我报告的保真度 东北项目评估中心(NEPEC)。VHA跟踪住院结果 时间到了。我们将招募32个团队参加基于电话的评估和现场保真度 拜访经验丰富的忠诚度评估师。我们将根据VHA设施的类型对团队进行分层 对前一年的表现进行自我评估。电话和现场订单 评估将是平衡的，有单独的评估员，以减少潜在的偏见。我们 将检查保真度方法与类内相关性之间的一致性程度。我们会 使用回归建模检查预测有效性，以检查 保真度方法和改善的住院结果。对于我们的成本识别分析，我们 将使用混合(重复)方法比较三种评估方法的成本 MEASURES)模型方法。我们还将包括形成性评估，以通知未来 在VHA和其他地方传播忠诚度评估方法。

Background: National policy has dramatically increased the emphasis on implementing evidence-based mental health services to meet the needs of people with severe mental illness, and the VHA has made great strides at providing effective, community-based services. One of the cornerstones of the VHA approach is Mental Health Intensive Case Management (MHICM), a model that is based on one of the most well-defined and empirically supported approaches: assertive community treatment. Most recently, VHA policy shifts have resulted in a proposed set of uniform mental health services to ensure access to a standard set of high quality mental health services, such as MHICM, across the entire VHA. However, successful implementation of evidence-based practices on a broad scale requires psychometrically valid, yet practical, ways to assess and monitor degree of implementation (i.e., fidelity). Currently, the only rigorous method to monitor implementation is an on-site fidelity visit, which is a very time-intensive, expensive, and burdensome approach for both the assessor and the program. Objective: The primary objective of the proposed study is to examine the effectiveness of innovative and potentially cost-effective methods to ensure the quality of mental health services for the most severely disabled veterans with mental illness. Specifically, we will examine the reliability and concurrent validity, predictive validity, and costs associated with three different methods of fidelity assessment: self-report, phone-based remote assessment, and a "gold-standard" on-site fidelity assessment. Methods: Currently all MHICM teams (n=111) provide self-reported fidelity annually to the Northeast Program Evaluation Center (NEPEC). VHA tracks hospitalization outcomes over time. We will recruit 32 teams to participate in a phone-based assessment and an on-site fidelity visit with experienced fidelity assessors. We will stratify teams on type of VHA facility and previous year's performance on the self-assessment. The order of phone and on-site assessments will be counter-balanced, with separate assessors, to reduce potential bias. We will examine level of agreement between fidelity approaches with intraclass-correlations. We will examine predictive validity using regression modeling to examine the association between fidelity method and improved hospitalization outcomes. For our cost identification analysis, we will compare costs across the three methods of assessment, using a mixed (repeated measures) model approach. We will also include a formative evaluation to inform future dissemination of fidelity assessment methods in the VHA and elsewhere.