The Role of Theory in Understanding and Resolving the Reliability Crisis

一个给定的实证检验的可靠性是用来设计该检验的理论质量的产物。从测试变量X到结果变量Y，有太多的未知因素或相互竞争的因果路径，使得结果不可靠。如果在理论不明确的情况下重复假设检验，那么整个研究领域可能是不可靠的。这种情况可以解释在该领域缺乏共识和未能重现研究结果的原因。特别是在行为科学和社会科学中，由于人类互动和社会组织的复杂性，理论歧义尤为严重。这个计划的目标是解决理论在可重复性危机中的潜在作用。具体来说：(1)测试一个研究领域的理论歧义程度。(2)检验因果模型的计算机辅助比较是否能够识别该领域经验不可靠性的具体来源。(3)测试众包理论主张是否可以提高该领域研究结果的可靠性和可复制性。(4)开发计算机系统，改进和节省这一过程，以便在认知、行为和社会科学领域部署测试和解决方案。前两个目标针对META-REP SPP提出的“为什么复制率不同”的问题，第三和第四个目标针对“如何提高复制率”的问题。正如我在提案中所讨论的，作为科学过程的一部分，复制和可靠性是密切相关的。因此，解决可靠性问题应该提高可复制性，如果不能帮助设计未来的复制，从而进一步推动科学的进步。该项目将首先在PI已知的单一领域测试拟议的方法（在第1-4点中开发）：宏观比较移民和再分配社会政策偏好。这个特殊的领域非常适合测试理论是否可以解释结果和重复的不可靠性，因为它的结果特别不可靠（相互竞争），并且主要依赖于容易获得的公共调查数据源。这个测试区域将使计算机系统的发展能够分析、存储和公开访问因果模型比较的结果。特定领域的结果也将导致该系统的进一步发展，从而使所提出的方法能够在认知、行为和社会科学的任何领域比较因果模型。

The reliability of a given empirical test is a product of the quality of the theory used to design that test. Too many unknowns or competing causal pathways from test variable X to outcome variable Y render the results unreliable. If hypothesis tests are repeated under theoretical ambiguity, an entire area of study may be unreliable. Such a scenario could explain a lack of consensus and failures to reproduce findings in that area. In the behavioral and social sciences especially, theoretical ambiguity is particularly acute due to the complexities of human interaction and societal organization. The goals of this proposed project are to address the potential role of theory in the reproducibility crisis. Specifically:(1) Test the extent of theoretical ambiguity in an area of study. (2) Test if computer-assisted comparison of causal models can identify the specific sources of empirical unreliability in that area. (3) Test if crowdsourcing theoretical claims can improve reliability and the replicability of findings in that area.(4) Develop computer systems that improve and economize this process so that tests and solutions can be deployed across cognitive, behavioral and social sciences.The first two goals target the question of “why replication rates vary” posed in the META-REP SPP. The third and fourth target the question of “how to increase replication rates”. As I discuss in the proposal, replication and reliability are closely connected as part of the scientific process. Therefore, addressing reliability issues should improve replicability, if not help design future replications and thus further the advancement of science.The project will first test the proposed methods (developed in points 1-4) in a single area known to the PI: macro-comparative immigration and redistributive social policy preferences. This particular area is ideally suited to testing whether theory can explain unreliability of results and replications because it has particularly unreliable (competing) results and relies mostly on easily accessible public survey data sources . This testing area will enable development of the computer system for analyzing, storing and publicly accessing the results of causal model comparisons. The area-specific results will also lead to the further development of this system so that it can enable the proposed methods of comparing causal models across any area of the cognitive, behavioral and social sciences.