The Role of Theory in Understanding and Resolving the Reliability Crisis

给定经验检验的可靠性是用于设计该检验的理论质量的产物。从检验变量X到结果变量Y，太多的未知数或相互竞争的因果路径使结果不可靠。如果假设检验在理论模棱两可的情况下反复进行，那么整个研究领域都可能是不可靠的。这种情况可以解释为什么在这一领域缺乏协商一致意见和未能复制调查结果。特别是在行为科学和社会科学中，由于人类互动和社会组织的复杂性，理论上的模棱两可尤其严重。这个拟议项目的目标是解决理论在可再生性危机中的潜在作用。具体地说：(1)检验某一研究领域的理论模糊程度。(2)检验计算机辅助的因果模型比较是否可以确定该领域经验不可靠的具体来源。(3)测试众包理论主张是否可以提高该领域研究结果的可靠性和可复制性。(4)开发计算机系统，改进和节约这一过程，以便测试和解决方案可以在认知、行为和社会科学中部署。前两个目标针对的是元代表SPP中提出的“为什么复制率不同”的问题。第三和第四条针对的是“如何提高复制率”的问题。正如我在提案中讨论的那样，作为科学过程的一部分，复制和可靠性密切相关。因此，解决可靠性问题应该提高可复制性，如果不能帮助设计未来的复制，从而进一步促进科学的进步。该项目将首先在PI已知的一个单一领域测试拟议的方法(在第1-4点开发)：宏观比较移民和再分配社会政策偏好。这一特定领域非常适合测试理论是否能够解释结果和复制的不可靠性，因为它的结果特别不可靠(相互竞争)，并且主要依赖于容易获得的公共调查数据来源。这一测试领域将使计算机系统的开发成为可能，以便分析、存储和公开获取因果模型比较的结果。特定领域的结果也将导致这一系统的进一步发展，从而使所提出的方法能够比较认知、行为和社会科学的任何领域的因果模型。

The reliability of a given empirical test is a product of the quality of the theory used to design that test. Too many unknowns or competing causal pathways from test variable X to outcome variable Y render the results unreliable. If hypothesis tests are repeated under theoretical ambiguity, an entire area of study may be unreliable. Such a scenario could explain a lack of consensus and failures to reproduce findings in that area. In the behavioral and social sciences especially, theoretical ambiguity is particularly acute due to the complexities of human interaction and societal organization. The goals of this proposed project are to address the potential role of theory in the reproducibility crisis. Specifically:(1) Test the extent of theoretical ambiguity in an area of study. (2) Test if computer-assisted comparison of causal models can identify the specific sources of empirical unreliability in that area. (3) Test if crowdsourcing theoretical claims can improve reliability and the replicability of findings in that area.(4) Develop computer systems that improve and economize this process so that tests and solutions can be deployed across cognitive, behavioral and social sciences.The first two goals target the question of “why replication rates vary” posed in the META-REP SPP. The third and fourth target the question of “how to increase replication rates”. As I discuss in the proposal, replication and reliability are closely connected as part of the scientific process. Therefore, addressing reliability issues should improve replicability, if not help design future replications and thus further the advancement of science.The project will first test the proposed methods (developed in points 1-4) in a single area known to the PI: macro-comparative immigration and redistributive social policy preferences. This particular area is ideally suited to testing whether theory can explain unreliability of results and replications because it has particularly unreliable (competing) results and relies mostly on easily accessible public survey data sources . This testing area will enable development of the computer system for analyzing, storing and publicly accessing the results of causal model comparisons. The area-specific results will also lead to the further development of this system so that it can enable the proposed methods of comparing causal models across any area of the cognitive, behavioral and social sciences.