The Reliability of Survey Data: The Effects of Question Content, Context and Form

每年收集大量的调查数据用于多种目的，包括收集政府使用的信息，民意和选举调查，广告和市场研究，以及社会和经济科学的基础研究。由于调查被广泛使用，我们必须建立一个关于调查数据的测量质量和产生最佳数据的问题属性的知识库。科学很好地理解了多项目措施的可靠性，或规模，通常用于此类研究。 但我们对单项调查问题的质量知之甚少。新的数据资源和建模策略现在可以评估这些调查措施的可靠性或一致性。该项目以评估测量可靠性的统计模型为基础，借鉴了调查中信息收集过程的见解。本研究的结果将提供切实可行的建议，将提高调查数据的质量，从而提高调查研究的普遍。该项目将开发一个公开网站，使所有调查研究人员能够调查大量常用调查问题的可靠性。 由于调查被科学界、政府和私营部门广泛用于为许多政策和经济决策提供信息，有关调查措施的改进信息将有助于美国人民的健康和福祉。S.由于调查被广泛使用，我们必须建立一个关于调查数据的测量质量和产生最佳结果的问题属性的知识库。为了实现这些目标，本项目开发了一套元数据，其中包含大约1200个单项问题的信息，这些问题代表了社会科学调查中使用的典型问题。调查问题的数据库，在十个国家的基础上开发，（或区域）代表性小组研究，包括一般社会调查、健康和退休研究、国家选举研究，包含对特定问题可靠性的估计，沿着问题属性的详细编码，如内容、回答格式和问题长度，其可用于相对于测量误差水平来评估调查问题的最佳属性。通过对大规模追踪调查问卷的信度信息和问卷属性的分析，本研究着重探讨了问卷构建的三大要素：内容、语境和形式。统计分析将涉及使用自回归或准马尔可夫单纯形建模技术。由此产生的元数据集将与大学间政治和社会研究联合会（ICPSR）和其他合适的数据档案馆存档。此外，该项目将把这些数据的关键要素连同所涉问题和问题背景的图像沿着张贴在一个公开网站上，使用户能够搜索、过滤或查询数据库，以调查感兴趣的各类调查问题的可靠性。这些发现将为调查方法的多学科领域提供重要信息，为社会学、政治学、经济学、传播学、心理学、法律、公共卫生和公共政策等学科的学者提供信息，该奖项反映了NSF的法定使命，并被认为值得通过使用基金会的智力价值进行评估来支持和更广泛的影响审查标准。

Vast amounts of survey data are collected each year for many purposes, including the collection of information for government use, public opinion and election surveys, advertising and marketing research, as well as basic research in the social and economic sciences. Because surveys are so widely used, it is essential that we establish a knowledge base about the measurement quality of survey data and the attributes of questions that produce the best data. Science well understands the reliability of multi-item measures, or scales, commonly used in such research. But we know much less about the quality of single-item survey questions. New data resources and modeling strategies now make it possible to assess the reliability or consistency of measurement for these survey measures. Based on statistical models for evaluating reliability of measurement, this project draws upon the insights from the information gathering process in surveys. The results of this research will provide practical suggestions that will improve the quality of survey data, thus improving survey research generally. The project will develop a publicly available website to allow all survey researchers to investigate question reliability of a large number of commonly used survey questions. Because surveys are used broadly by science, government and the private sector to inform many policy and economic decisions, improved information about survey measures will contribute to the health and well being of U. S. society across multiple sectors and over time.Because surveys are so widely used, it is essential that we establish a knowledge base about the measurement quality of survey data and the attributes of questions that produce the best results. In order to accomplish these aims, this project develops a set of meta-data containing information for roughly 1200 single-item questions representative of typical questions used in social science surveys. The database of survey questions, developed on the basis of ten nationally (or regionally) representative panel studies including the General Social Surveys, the Health and Retirement Study, the National Election Studies, contains estimates of question-specific reliabilities, along with detailed coding of attributes of the questions, such as their content, response formats, and question length, which can be used to evaluate the optimal properties of survey questions with respect to levels of measurement error. Through an analysis of the reliability information and the attributes of survey questions from these large-scale panel studies, the study focuses on three major elements of questionnaire construction: content, context and form. Statistical analysis will involve use of auto-regressive or Quasi-Markov simplex modeling techniques. The resulting meta-data set will be archived with the Inter-university Consortium for Political and Social Research (ICPSR) and other suitable data archives. In addition, the project will post key elements of these data on a publicly available website, along with images of questions and question context involved, in a manner that allows users to search, filter or query the data base in investigating the reliability of types of survey questions of interest. These findings will contribute important information to the multidisciplinary field of survey methodology, which informs scholars in disciplines including sociology, political science, economics, communication, psychology, law, public health, and public policy, as well as actors in the private and governmental sectors.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.