RR: Establishing and Boosting Confidence Levels for Empirical Research Using Twitter Data

在学术界和广大公众中，对科学研究重现性危机的担忧日益普遍。对科学本身进行科学研究的元科学领域正在蓬勃发展，并审查了对可重复和稳健研究的威胁的存在和普遍存在。然而，社会科学中现有的大多数复制努力都集中在使用统计上严格设计的调查或实验的数据进行的研究。致力于利用有机数据，包括无处不在的传感器或移动应用程序、推特订阅源、点击流等有机产生的数据，审查社会科学学术出版物中对有机数据的不一致处理做法的复制工作在很大程度上缺失。由于社会和行为科学的发现为政策制定者提供了从国土安全到国民经济等广泛问题的信息，因此建立对这些发现的信心对于正确使用这些发现至关重要，因此对所有这些国家优先应用领域都有更广泛的影响。更具体地说，这个项目从使用既不能重现也不能推广的有机数据来确定实证研究的范围、原因和补救措施开始。这一步骤的发现提高了人们对社会科学许多领域收集、清理和处理有机数据集的标准和工具的认识。此外，该项目还开发了新的分析框架和方法，可用于评估有机数据经验研究的可重复性和稳健性。我们的愿景是将这种框架广泛应用于许多应用领域，从而促进社会科学不同领域的文化变革，并将可重复性和稳健性的价值带入数据密集型研究的前沿。该奖项反映了NSF的法定使命，并通过使用基金会的智力优势和更广泛的影响审查标准进行评估，被认为值得支持。

Concerns about a reproducibility crisis in scientific research have become increasingly prevalent within the academic community and to the public at large. The field of meta-science, which performs the scientific study of science itself, is thriving and has examined the existence and prevalence of threats to reproducible and robust research. Most existing replication efforts in social sciences, however, have focused on studies using data from statistically rigorous designed surveys or experiments. Largely missing are replication efforts devoted to examining those studies with organic data, including data organically generated by ubiquitous sensors or mobile applications, twitter feeds, click streams, etc. This project examines the inconsistent handling practices of organic data among scholarly publications in social sciences, in order to establish the confidence (or the lack thereof) in the conclusions drawn from such data analysis. Since findings of social and behavioral sciences inform policy makers on a wide variety of issues, from homeland security to national economy, establishing the confidence of these findings is critical for the proper usage of them, and therefore has broader impacts on all these application areas of national priority.More specifically, this project starts with determining the extent of, causes of, and remedies for empirical research using organic data that are neither reproducible nor generalizable. The findings from this step raise awareness about the standards and tools for collecting, cleaning, and processing organic data sets across many fields of social sciences. In addition, this project develops new analytical frameworks and methodologies useful for evaluating replicability and robustness of empirical studies with organic data. The vision is for such frameworks to be broadly used in many application domains, thereby fostering cultural change across different fields in social sciences, and bringing the value of reproducibility and robustness to the forefront of data intensive research.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.