The Role of Theory in Understanding and Resolving the Reliability Crisis

点击翻译按钮获取中文摘要

The reliability of a given empirical test is a product of the quality of the theory used to design that test. Too many unknowns or competing causal pathways from test variable X to outcome variable Y render the results unreliable. If hypothesis tests are repeated under theoretical ambiguity, an entire area of study may be unreliable. Such a scenario could explain a lack of consensus and failures to reproduce findings in that area. In the behavioral and social sciences especially, theoretical ambiguity is particularly acute due to the complexities of human interaction and societal organization. The goals of this proposed project are to address the potential role of theory in the reproducibility crisis. Specifically:(1) Test the extent of theoretical ambiguity in an area of study. (2) Test if computer-assisted comparison of causal models can identify the specific sources of empirical unreliability in that area. (3) Test if crowdsourcing theoretical claims can improve reliability and the replicability of findings in that area.(4) Develop computer systems that improve and economize this process so that tests and solutions can be deployed across cognitive, behavioral and social sciences.The first two goals target the question of “why replication rates vary” posed in the META-REP SPP. The third and fourth target the question of “how to increase replication rates”. As I discuss in the proposal, replication and reliability are closely connected as part of the scientific process. Therefore, addressing reliability issues should improve replicability, if not help design future replications and thus further the advancement of science.The project will first test the proposed methods (developed in points 1-4) in a single area known to the PI: macro-comparative immigration and redistributive social policy preferences. This particular area is ideally suited to testing whether theory can explain unreliability of results and replications because it has particularly unreliable (competing) results and relies mostly on easily accessible public survey data sources . This testing area will enable development of the computer system for analyzing, storing and publicly accessing the results of causal model comparisons. The area-specific results will also lead to the further development of this system so that it can enable the proposed methods of comparing causal models across any area of the cognitive, behavioral and social sciences.