RR: Establishing and Boosting Confidence Levels for Empirical Research Using Twitter Data

对科学研究中再现性危机的担忧在学术界和广大公众中越来越普遍。 对科学本身进行科学研究的元科学领域正在蓬勃发展，并审查了对可重复和强大研究的威胁的存在和普遍性。 然而，社会科学中大多数现有的复制工作都集中在使用统计上严格设计的调查或实验数据的研究上。大部分缺失的是复制的努力，致力于研究这些研究与有机数据，包括无处不在的传感器或移动的应用程序，twitter的饲料，点击流等有机产生的数据。这个项目研究了不一致的处理方法，在社会科学的学术出版物之间的有机数据，以建立信心（或缺乏）从这样的数据分析得出的结论。 由于社会科学和行为科学的研究成果为决策者提供了从国土安全到国民经济等各种问题的信息，因此建立对这些研究成果的信心对于正确使用它们至关重要，因此对国家优先考虑的所有这些应用领域都有更广泛的影响。更具体地说，本项目从确定和补救措施的实证研究，使用有机数据，既不是可复制的，也不是普遍的。 这一步骤的发现提高了人们对收集，清理和处理社会科学许多领域有机数据集的标准和工具的认识。此外，该项目还开发了新的分析框架和方法，有助于评估有机数据实证研究的可复制性和稳健性。该奖项旨在使这些框架广泛应用于许多应用领域，从而促进社会科学不同领域的文化变革，并将可重复性和鲁棒性的价值带到数据密集型研究的最前沿。该奖项反映了NSF的法定使命，并通过使用基金会的知识价值和更广泛的影响审查标准进行评估，被认为值得支持。

Concerns about a reproducibility crisis in scientific research have become increasingly prevalent within the academic community and to the public at large. The field of meta-science, which performs the scientific study of science itself, is thriving and has examined the existence and prevalence of threats to reproducible and robust research. Most existing replication efforts in social sciences, however, have focused on studies using data from statistically rigorous designed surveys or experiments. Largely missing are replication efforts devoted to examining those studies with organic data, including data organically generated by ubiquitous sensors or mobile applications, twitter feeds, click streams, etc. This project examines the inconsistent handling practices of organic data among scholarly publications in social sciences, in order to establish the confidence (or the lack thereof) in the conclusions drawn from such data analysis. Since findings of social and behavioral sciences inform policy makers on a wide variety of issues, from homeland security to national economy, establishing the confidence of these findings is critical for the proper usage of them, and therefore has broader impacts on all these application areas of national priority.More specifically, this project starts with determining the extent of, causes of, and remedies for empirical research using organic data that are neither reproducible nor generalizable. The findings from this step raise awareness about the standards and tools for collecting, cleaning, and processing organic data sets across many fields of social sciences. In addition, this project develops new analytical frameworks and methodologies useful for evaluating replicability and robustness of empirical studies with organic data. The vision is for such frameworks to be broadly used in many application domains, thereby fostering cultural change across different fields in social sciences, and bringing the value of reproducibility and robustness to the forefront of data intensive research.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.