Comparison of Fidelity Assessment Methods

背景：国家政策大大加强了对执行 以证据为基础的精神卫生服务，以满足严重精神疾病患者的需求， VHA在提供有效的社区服务方面取得了很大的进步。之一 VHA方法的基石是心理健康强化病例管理（MHICM）， 一个基于最明确定义和经验支持的方法之一的模型： 自信的社区治疗。最近，VHA政策的转变导致了一套拟议的 统一的精神卫生服务，以确保获得一套标准的高质量精神卫生服务。 卫生服务，如MHICM，在整个VHA。然而，成功实施 大规模的循证实践需要心理测量学上有效但实用的方法， 评估和监测执行程度（即，fidelity）。目前，唯一严格的 监测执行情况的方法是现场实地访问，这是一个非常耗时， 对于评估员和项目来说，这种方法既昂贵又繁琐。 目的：本研究的主要目的是检查 创新和潜在的成本效益的方法，以确保精神卫生的质量 为患有精神疾病的最严重残疾退伍军人提供服务。具体来说，我们将 检查可靠性和并发效度，预测效度，以及与 三种不同的保真度评估方法：自我报告，基于电话的远程评估， 和“黄金标准”的现场保真度评估。 方法：目前，所有MHICM团队（n=111）每年提供自我报告的保真度， 东北计划评估中心（NEPEC）。VHA跟踪住院结局， 时间我们将招募32个团队参加电话评估和现场保真 与经验丰富的忠诚度评估员一起访问。我们将根据VHA设施的类型对团队进行分层， 上一年的自我评价。电话和现场订购 评估将由单独的评估员进行平衡，以减少潜在的偏倚。我们 将检查保真度方法与类内相关性之间的协议水平。我们将 使用回归模型检查预测有效性，以检查 保真度方法和改善的住院治疗结果。对于成本识别分析，我们 将使用混合（重复）评估方法比较三种评估方法的成本 （三）示范方法。我们还将包括形成性评估，以告知未来 在VHA和其他地方传播保真度评估方法。

Background: National policy has dramatically increased the emphasis on implementing evidence-based mental health services to meet the needs of people with severe mental illness, and the VHA has made great strides at providing effective, community-based services. One of the cornerstones of the VHA approach is Mental Health Intensive Case Management (MHICM), a model that is based on one of the most well-defined and empirically supported approaches: assertive community treatment. Most recently, VHA policy shifts have resulted in a proposed set of uniform mental health services to ensure access to a standard set of high quality mental health services, such as MHICM, across the entire VHA. However, successful implementation of evidence-based practices on a broad scale requires psychometrically valid, yet practical, ways to assess and monitor degree of implementation (i.e., fidelity). Currently, the only rigorous method to monitor implementation is an on-site fidelity visit, which is a very time-intensive, expensive, and burdensome approach for both the assessor and the program. Objective: The primary objective of the proposed study is to examine the effectiveness of innovative and potentially cost-effective methods to ensure the quality of mental health services for the most severely disabled veterans with mental illness. Specifically, we will examine the reliability and concurrent validity, predictive validity, and costs associated with three different methods of fidelity assessment: self-report, phone-based remote assessment, and a "gold-standard" on-site fidelity assessment. Methods: Currently all MHICM teams (n=111) provide self-reported fidelity annually to the Northeast Program Evaluation Center (NEPEC). VHA tracks hospitalization outcomes over time. We will recruit 32 teams to participate in a phone-based assessment and an on-site fidelity visit with experienced fidelity assessors. We will stratify teams on type of VHA facility and previous year's performance on the self-assessment. The order of phone and on-site assessments will be counter-balanced, with separate assessors, to reduce potential bias. We will examine level of agreement between fidelity approaches with intraclass-correlations. We will examine predictive validity using regression modeling to examine the association between fidelity method and improved hospitalization outcomes. For our cost identification analysis, we will compare costs across the three methods of assessment, using a mixed (repeated measures) model approach. We will also include a formative evaluation to inform future dissemination of fidelity assessment methods in the VHA and elsewhere.