The Role of Theory in Understanding and Resolving the Reliability Crisis

理论在理解和解决可靠性危机中的作用

• 批准号: 464546557
• 负责人: Dr. Nate Breznau, Ph.D.
• 金额: --
• 依托单位: Department of Sociology
• 依托单位国家: 德国
• 项目类别: Priority Programmes
• 资助国家: 德国
• 项目状态: 未结题
• 来源: https://gepris.dfg.de/gepris/projekt/464546557?language=en
• 关键词: Role; Theory; Understanding; Resolving; Reliability

The reliability of a given empirical test is a product of the quality of the theory used to design that test. Too many unknowns or competing causal pathways from test variable X to outcome variable Y render the results unreliable. If hypothesis tests are repeated under theoretical ambiguity, an entire area of study may be unreliable. Such a scenario could explain a lack of consensus and failures to reproduce findings in that area. In the behavioral and social sciences especially, theoretical ambiguity is particularly acute due to the complexities of human interaction and societal organization. The goals of this proposed project are to address the potential role of theory in the reproducibility crisis. Specifically:(1) Test the extent of theoretical ambiguity in an area of study. (2) Test if computer-assisted comparison of causal models can identify the specific sources of empirical unreliability in that area. (3) Test if crowdsourcing theoretical claims can improve reliability and the replicability of findings in that area.(4) Develop computer systems that improve and economize this process so that tests and solutions can be deployed across cognitive, behavioral and social sciences.The first two goals target the question of “why replication rates vary” posed in the META-REP SPP. The third and fourth target the question of “how to increase replication rates”. As I discuss in the proposal, replication and reliability are closely connected as part of the scientific process. Therefore, addressing reliability issues should improve replicability, if not help design future replications and thus further the advancement of science.The project will first test the proposed methods (developed in points 1-4) in a single area known to the PI: macro-comparative immigration and redistributive social policy preferences. This particular area is ideally suited to testing whether theory can explain unreliability of results and replications because it has particularly unreliable (competing) results and relies mostly on easily accessible public survey data sources . This testing area will enable development of the computer system for analyzing, storing and publicly accessing the results of causal model comparisons. The area-specific results will also lead to the further development of this system so that it can enable the proposed methods of comparing causal models across any area of the cognitive, behavioral and social sciences.

一个给定的实证检验的可靠性是用于设计该检验的理论质量的产物。从测试变量X到结果变量Y，有太多的未知数或竞争性因果路径会使结果不可靠。如果假设检验是在理论模糊的情况下重复进行的，那么整个研究领域都可能是不可靠的。这种情况可以解释为什么在这一领域缺乏共识和未能复制调查结果。特别是在行为科学和社会科学中，由于人类互动和社会组织的复杂性，理论上的模糊性特别严重。这个项目的目标是解决理论在再现性危机中的潜在作用。具体而言：（1）测试研究领域中理论模糊的程度。(2)测试计算机辅助的因果模型比较是否可以识别该领域经验不可靠性的具体来源。(3)测试众包理论主张是否可以提高该领域研究结果的可靠性和可复制性。(4)开发计算机系统来改进和节省这个过程，以便测试和解决方案可以在认知，行为和社会科学中部署。前两个目标针对META-REP SPP中提出的“为什么复制率不同”的问题。第三和第四个目标是“如何提高复制率”的问题。正如我在提案中所讨论的那样，复制和可靠性作为科学过程的一部分密切相关。因此，解决可靠性问题应该提高可复制性，如果没有帮助设计未来的复制，从而进一步的科学进步。该项目将首先测试所提出的方法（在点1-4）在一个单一的领域已知的PI：宏观比较移民和再分配的社会政策偏好。这一特定领域非常适合测试理论是否可以解释结果和重复的不可靠性，因为它具有特别不可靠的（竞争性）结果，并且主要依赖于容易获得的公共调查数据来源。这个试验区将使计算机系统的开发，用于分析，存储和公开访问因果模型比较的结果。具体地区的结果也将导致这一系统的进一步发展，使它能够使拟议的方法比较因果模型 在认知、行为和社会科学的任何领域。