The Reliability of Survey Data: The Effects of Question Content, Context and Form

每年收集大量的调查数据，用于多种目的，包括收集供政府使用的信息、民意和选举调查、广告和营销研究以及社会和经济科学的基础研究。由于调查被如此广泛地使用，我们有必要建立一个关于调查数据的测量质量和产生最佳数据的问题属性的知识库。科学很好地理解了这类研究中常用的多项目测量或量表的可靠性。但我们对单项调查问题的质量知之甚少。现在，新的数据资源和建模策略使评估这些调查测量的可靠性或一致性成为可能。本项目以评估测量可靠性的统计模型为基础，借鉴了调查中信息收集过程的见解。本研究的结果将提供切实可行的建议，提高调查数据的质量，从而提高调查研究的总体水平。该项目将开发一个公开可用的网站，让所有调查研究人员调查大量常用调查问题的问题可靠性。由于科学、政府和私营部门广泛使用调查来为许多政策和经济决策提供信息，因此，随着时间的推移，有关调查措施的信息的改进将有助于美国社会在多个部门的健康和福祉。由于调查被如此广泛地使用，我们有必要建立一个关于调查数据的测量质量和产生最佳结果的问题属性的知识库。为了实现这些目标，本项目开发了一组元数据，其中包含大约1200个单项问题的信息，这些问题代表了社会科学调查中使用的典型问题。调查问题数据库是根据十个具有全国（或区域）代表性的小组研究（包括一般社会调查、健康和退休研究、全国选举研究）编制的，其中载有对特定问题可靠性的估计，以及对问题属性的详细编码，例如问题的内容、回答格式和问题长度。这可以用来评估关于测量误差水平的调查问题的最佳属性。通过分析这些大规模面板研究的信度信息和调查问题的属性，本研究重点研究了问卷构建的三个主要要素：内容、语境和形式。统计分析将涉及使用自回归或拟马尔可夫单纯形建模技术。由此产生的元数据集将与大学间政治和社会研究联盟（ICPSR）和其他适当的数据档案一起存档。此外，该项目将把这些数据的关键要素连同所涉及的问题和问题背景的图像一起公布在一个公开可用的网站上，以便用户在调查感兴趣的调查问题类型的可靠性时搜索、过滤或查询数据库。这些发现将为调查方法的多学科领域提供重要信息，为社会学、政治学、经济学、传播学、心理学、法律、公共卫生和公共政策等学科的学者以及私营和政府部门的行动者提供信息。该奖项反映了美国国家科学基金会的法定使命，并通过使用基金会的知识价值和更广泛的影响审查标准进行评估，被认为值得支持。

Vast amounts of survey data are collected each year for many purposes, including the collection of information for government use, public opinion and election surveys, advertising and marketing research, as well as basic research in the social and economic sciences. Because surveys are so widely used, it is essential that we establish a knowledge base about the measurement quality of survey data and the attributes of questions that produce the best data. Science well understands the reliability of multi-item measures, or scales, commonly used in such research. But we know much less about the quality of single-item survey questions. New data resources and modeling strategies now make it possible to assess the reliability or consistency of measurement for these survey measures. Based on statistical models for evaluating reliability of measurement, this project draws upon the insights from the information gathering process in surveys. The results of this research will provide practical suggestions that will improve the quality of survey data, thus improving survey research generally. The project will develop a publicly available website to allow all survey researchers to investigate question reliability of a large number of commonly used survey questions. Because surveys are used broadly by science, government and the private sector to inform many policy and economic decisions, improved information about survey measures will contribute to the health and well being of U. S. society across multiple sectors and over time.Because surveys are so widely used, it is essential that we establish a knowledge base about the measurement quality of survey data and the attributes of questions that produce the best results. In order to accomplish these aims, this project develops a set of meta-data containing information for roughly 1200 single-item questions representative of typical questions used in social science surveys. The database of survey questions, developed on the basis of ten nationally (or regionally) representative panel studies including the General Social Surveys, the Health and Retirement Study, the National Election Studies, contains estimates of question-specific reliabilities, along with detailed coding of attributes of the questions, such as their content, response formats, and question length, which can be used to evaluate the optimal properties of survey questions with respect to levels of measurement error. Through an analysis of the reliability information and the attributes of survey questions from these large-scale panel studies, the study focuses on three major elements of questionnaire construction: content, context and form. Statistical analysis will involve use of auto-regressive or Quasi-Markov simplex modeling techniques. The resulting meta-data set will be archived with the Inter-university Consortium for Political and Social Research (ICPSR) and other suitable data archives. In addition, the project will post key elements of these data on a publicly available website, along with images of questions and question context involved, in a manner that allows users to search, filter or query the data base in investigating the reliability of types of survey questions of interest. These findings will contribute important information to the multidisciplinary field of survey methodology, which informs scholars in disciplines including sociology, political science, economics, communication, psychology, law, public health, and public policy, as well as actors in the private and governmental sectors.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.