Evaluation-Based Effectiveness Testing

描述（由申请人提供）：学校，社区和服务提供商开始实施具有证明或有前途的潜力的计划。作为传播工作的一部分，往往要求当地执行者进行当地评价。然而，这些评价往往不够有力，缺乏适当的或任何比较组，并受到衰减的有效性估计，由于不可靠的措施或样本的不变性。该SBIR的目标是开发一种工具，帮助当地评估人员和政策制定者获得有效、可靠和有力的证据，以证明其干预措施的有效性，而不必承担识别、召集、维护和测量当地比较组的负担。该系统将通过元分析方式结合联合收割机先前评价的存档数据，包括相关项目干预组和对照组的评价数据，为当地评价人员建立适当的反事实。将制定战略，使用户能够完成有助于解释当地评价结果的分析。因此，而不是依赖于一个典型的无法获得的金标准的方法来评估影响（随机对照试验），这种方法的目标将是提供一个统计上可靠的方法来确定当地的程序性能，并提供程序有用的基准，对当地的评估结果可以解释。该项目将利用先前由SBIR基金开发的现有资源，帮助学校和社区进行地方评估。这个现有的系统，评估LIFE，现在是一个活跃的服务，并在过去几年中积累了一个大型数据库，其中不仅包括行为结果的调查数据，与结果相关的调解人，以及调查参与者的人口统计信息，而且还将这些数据与正在实施的方案和采用的评估设计联系起来。在这个第一阶段的项目中，我们将：1）我们将收集现存的数据集，这些数据集适合完成拟议方法的试点测试，并确保数据被编码，以提供适当的结果池，用于完成荟萃分析分析; 2）完善荟萃分析统计方法，用于使用参考数据完成比较;以及3）使用来自五到十个地方评估的数据来测试这些方法，以开发并随后展示所提出的方法对地方项目评估者、服务提供者和资助者的效用。

DESCRIPTION (provided by applicant): Schools, communities, and service providers are beginning to implement programs with proven or promising potential. As part of this dissemination effort, local implementers are often required to conduct local evaluations. These evaluations, however, are often underpowered, lack appropriate or any comparison group, and are subject to attenuated effectiveness estimates due to unreliable measures or sample invariance. The goal of this SBIR is to develop a tool to help local evaluators and policy makers obtain valid, reliable, and robust evidence for the effectiveness of their intervention practices without the burden of having to identify convene, maintain, and measure a local comparison group. The system will meta-analytically combine archived data from prior evaluations, including evaluation data from intervention groups of relevant programs and from control groups to establish appropriate counterfactuals for local evaluators. Strategies will be developed that will allow users to complete analyses that will useful for interpreting local evaluation outcomes. Thus, rather than relying on a typically unobtainable gold-standard method for assessing impact (randomized control trials), the goal of this approach will be to provide a statistically reliable method for determining local program performance and to provide programs useful benchmarks against which local evaluation findings can be interpreted. This project will take advantage of an existing resource, previously developed with SBIR funds that assists schools and communities in performing local evaluations. This existing system, Evaluation Lizard, is now an active service and has, during the past several years amassed a large database that includes not only survey data on behavioral outcomes, mediators related to outcomes, and demographic information about survey participants, but also ties these data to programs being implemented and the evaluation design that was adopted. In this Phase I project we will: 1) We will assemble extant datasets that are appropriate for completing a pilot test of proposed methods and ensure that data are coded to provide appropriate pools of results for completing meta-analytic analyses; 2) refine meta-analytic statistical methods for completing comparisons using referent data; and 3) test these methods using data from five to ten local evaluations to develop and then demonstrate the utility of the proposed methods for local program evaluators, service providers, and funders.