Measuring the Quality of Delivery of Prevention

描述（由申请人提供）：众所周知，药物滥用预防计划的有效性取决于交付的质量。随着以研究为基础的项目的广泛传播，拥有评估交付质量的工具对于记录和理解项目成功和失败的条件变得非常重要。目前还没有标准的方法来衡量预防项目的质量。这个项目的目标是创建一个标准化的工具来评估以研究为基础的药物滥用预防项目的交付质量。在商业上，将该产品添加到现有产品（评估蜥蜴）中，该产品提供针对基于证据的项目评估量身定制的测试前和测试后调查，将大大加强我们在预防评估市场的地位。该SBIR项目的广泛目标是开发一种工具，可以收集广泛的项目的交付质量的标准化措施，这些项目由于被列入NREPP模型项目清单而被传播，并且新开发的项目希望有资格在未来被纳入。在第一阶段，我们采访了预防研究人员、项目管理人员、项目开发人员和政策制定者，以定义和详细说明定义实施质量维度的基本标准。基于他们的输入，我们设计了一个原型系统，使我们能够完成以下任务：(1)为选定的NREPP辅导员提供的预防项目创建会话和项目评分表单模板，(2)将模板链接到数据库以跟踪表单信息（项目、会话、观察员、教师和创建日期），以及(3)创建远程打印的PDF表单，可以通过条形码链接回数据库。原型系统测量剂量、依从性、参与度和适应性。我们还编写了一份使用该系统的手册。虽然该系统可以用于中试，但它还没有完全自动化，也没有包含许多元素。在第二阶段，我们的首要任务将是完成递送质量数据收集系统，用于收集递送质量数据，该系统将包括评估依从性、适应性、剂量、参与和总体质量测量的工具。我们将开发在线应用程序，将收集到的有关课程交付质量的数据与学生的测试前-测试后结果数据以各种方式联系起来，并进行联合分析。为了确保应用程序按预期工作，我们将对所有组件应用程序进行alpha和beta测试。我们将开发一个在线系统，用于进行基本分析和准备报告，以满足五个不同专业群体的需求：项目执行者、项目管理者、项目开发者、预防研究人员和政策制定者。我们将在10个预防机构和学区进行实地试验，以证明交付数据收集和报告系统的质量为项目执行者、预防机构管理者和主管以及政府机构预防管理者提供了有助于记录和理解预防项目交付过程的信息。已建立的预防机构将采用评估蜥蜴系统，包括测试前-测试后调查和交付质量报告表。我们假设，参与小组将报告对该系统的运作感到满意，并将报告对如何在其职权范围内实施预防的认识有所提高。此外，我们将进行一项信度和效度子研究，在该研究中，参与机构将被要求选择一名教师视频记录他们所教的课程，以便观察数据可以纳入分析，以证明措施的信度和效度。

DESCRIPTION (provided by applicant): Quality of delivery is known to moderate the effectiveness of drug abuse prevention programs. As research-based programs become widely disseminated, having tools for assessing quality of delivery becomes important for documenting and understanding the conditions under which programs succeed and fail. There are currently no standard methods for measuring the quality with which prevention programs are delivered. The goal of this project is to create a standardized tool for assessing quality of delivery for research-based drug abuse prevention programs.. Commercially, the addition of this product to an existing product (Evaluation Lizard) that provides pretest and posttest surveys tailored to the evaluation of evidence-based programs, will significantly strengthen our niche in the prevention evaluation marketplace. The broad aim of this SBIR project is to develop a tool that will allow the collection of standardized measures of quality of delivery across a wide range of programs being disseminated as a result of their inclusion on the NREPP model programs list and newly developed programs that wish to qualify for future inclusion. During Phase I, we interviewed prevention researchers, program administrators, program developers, and policy makers to define and elaborate on the essential criteria for defining dimensions of quality of implementation. Based on their input, we designed a prototype system that allowed us to accomplish the following tasks: (1) create session and program rating form templates for selected NREPP facilitator-delivered prevention programs, (2) link templates to a database to track form information (program, session, observer, teacher, and date of creation), and (3) create PDF forms printed remotely that can be linked by bar code back to the database. The prototype system measures dosage, adherence, engagement, and adaptation. We also developed a manual for using the system. While functional for the pilot test, this system is not yet fully automated and does not include numerous elements. During Phase II, our first task will be to complete the quality of delivery data gathering system for collecting quality of delivery data that will include tools for assessing adherence, adaptation, dosage, engagement, and measures of overall quality. We will develop online applications that will allow data collected about quality of program delivery to be linked and jointly analyzed in various ways with pretest-posttest outcome data from students. To ensure that the application works as intended, we will conduct alpha and beta tests of all component applications. We will develop an online system for conducting basic analysis and preparing reports that meets the needs of five separate professional groups: program implementers, program administrators, program developers, prevention researchers, and policy makers. We will conduct a field trial with 10 prevention agencies and school districts to demonstrate that the quality of delivery data collection and reporting system provide program implementers, prevention agency administrators and supervisors, and government agency prevention administrators with information useful to documenting and understanding the process of prevention program delivery. Established prevention agencies will adopt the use of the Evaluation Lizard system, including both pretest-posttest surveys and quality of delivery report forms. We hypothesize that participating groups will report satisfaction with the functioning of the system and will report increased awareness of how prevention is being implemented in their area of purview. Further, we will conduct a reliability and validity sub-study in which participating agencies will be asked to select one teacher to video record the lessons they teach so that observation data can be included in the analysis in order to demonstrate reliability and validity of the measures. PUBLIC HEALTH RELEVANCE: The field of drug abuse prevention research has identified numerous interventions that can be effective; however, unless programs are implemented well in practice, they are unlikely to achieve their desired effect, and the effort and expense of these programs may be wasted. The need to assess quality of implementation is widely recognized, but, to date, there is no unified strategy for gathering or reporting about data. The value of the product we are proposing is that it would provide an easy-to-use system for documenting quality of program implementation that could be linked to outcome data.