"Improved Methodologies for Field Experiments: Maximizing Statistical Power While Promoting Replication."

“改进现场实验方法：在促进复制的同时最大化统计能力。”

• 批准号: 1461491
• 负责人: Michael Anderson
• 金额: $ 46万
• 依托单位: University of California-Berkeley
• 依托单位国家: 美国
• 项目类别: Standard Grant
• 财政年份: 2015
• 资助国家: 美国
• 起止时间: 2015-09-01 至 2019-08-31
• 项目状态: 已结题
• 来源: https://www.nsf.gov/awardsearch/showAward?AWD_ID=1461491&HistoricalAwards=false
• 关键词: Improved; Methodologies; Field; Experiments; Maximizing

This project proposes new methods for analyzing data from randomized controlled trials (RCTs), which are now increasingly used in some areas of economics and other social sciences. Researchers often want to use an RCT to test multiple different hypotheses. However, carrying out this kind of data analysis requires careful consideration of statistical issues; in particular, there is a chance that using the same data to run multiple different statistical analyses will increase the chance of false positives. As a result, economists conducting RCTs are increasingly specifying pre-analysis plans; that is, they are specifying how they plan to analyze the data before beginning the analysis. These plans, however, do limit researchers' ability to reuse data and learn from the data before beginning the statistical analysis. The research team will develop a potential mechanism for researchers to learn from the data without specifying every hypothesis in advance. They will use concepts from machine-learning as well as probability theory to provide a better framework for how to maximize power within a given RCT experiment. The results could therefore improve data analysis in important ways.The research team has evidence that in practice preanalysis plans include tests for many, even hundreds, of hypotheses. As a result current preanalysis plan designs are likely to be underpowered for many effects of economic interest. The team will extend and apply techniques from biostatistics, including gatekeeping, sequential testing, and FDR control to the design of preanalysis plans. They will also develop a split-sample approach in which researchers conduct exploratory analysis on one part of the data set, and, using estimates from that part of the data combined with theory over likely mechanisms, refine their hypotheses before registering a subset of hypotheses to be tested on the remaining part of the data.

该项目提出了分析随机对照试验（RCT）数据的新方法，这些方法现在越来越多地用于经济学和其他社会科学的某些领域。 研究人员通常希望使用RCT来测试多个不同的假设。 然而，进行这种数据分析需要仔细考虑统计问题;特别是，使用相同的数据进行多个不同的统计分析可能会增加误报的机会。 因此，进行随机对照试验的经济学家越来越多地指定分析前计划;也就是说，他们在开始分析之前指定他们计划如何分析数据。 然而，这些计划确实限制了研究人员在开始统计分析之前重用数据和从数据中学习的能力。 研究团队将开发一种潜在的机制，让研究人员从数据中学习，而无需事先指定每一个假设。 他们将使用机器学习和概率论的概念，为如何在给定的RCT实验中最大化功率提供更好的框架。 研究小组有证据表明，在实践中，预分析计划包括对许多甚至数百个假设的测试。 因此，目前的预分析计划设计可能对许多经济利益的影响动力不足。 该团队将扩展和应用生物统计学技术，包括守门，序贯检验和FDR控制，以设计分析前计划。他们还将开发一种分裂样本的方法，研究人员对数据集的一部分进行探索性分析，并使用该部分数据的估计值与可能机制的理论相结合，在注册一个子集的假设之前完善他们的假设，以在数据的其余部分进行测试。