The Reliability of Survey Data: The Effects of Question Content, Context and Form

每年收集的大量调查数据用于许多目的，包括收集政府使用的信息、民意和选举调查、广告和营销研究以及社会科学和经济学的基础研究。由于调查的应用如此广泛，因此我们必须建立一个关于调查数据的测量质量和产生最佳数据的问题的属性的知识库。科学很好地理解此类研究中常用的多项测量或标尺的可靠性。但我们对单项调查问题的质量知之甚少。新的数据来源和建模策略现在使评估这些调查措施的测量的可靠性或一致性成为可能。在评估测量可靠性的统计模型的基础上，本项目借鉴了调查中信息收集过程的见解。本研究的结果将为提高调查数据的质量提供切实可行的建议，从而提高调查研究的整体水平。该项目将开发一个公开可用的网站，使所有调查研究人员能够调查大量常用调查问题的问题可靠性。由于调查被科学、政府和私营部门广泛用于为许多政策和经济决策提供信息，有关调查措施的改进信息将有助于多个部门和时间段美国社会的健康和福祉。由于调查被如此广泛地使用，我们必须建立一个关于调查数据的衡量质量和产生最佳结果的问题的属性的知识库。为了实现这些目标，该项目开发了一组元数据，其中包含大约1200个单项问题的信息，这些问题代表了社会科学调查中使用的典型问题。调查问题数据库是在包括一般社会调查、健康和退休研究、国家选举研究在内的10个全国性(或地区性)代表性小组研究的基础上开发的，包含对问题具体可靠性的估计，以及问题属性的详细编码，如问题的内容、回答格式和问题长度，这些属性可用于评估调查问题相对于测量误差水平的最佳特性。通过对这些大规模小组研究中的调查问题的可信度信息和属性的分析，本研究着重于问卷构建的三个主要要素：内容、背景和形式。统计分析将涉及使用自回归或准马尔可夫单纯形建模技术。由此产生的元数据集将保存在大学间政治和社会研究联盟(政治和社会研究联合会)和其他适当的数据档案中。此外，该项目将在一个可公开使用的网站上张贴这些数据的关键要素，以及所涉问题和问题背景的图像，使用户能够在调查感兴趣的各类调查问题的可靠性时搜索、筛选或查询数据库。这些发现将为调查方法的多学科领域提供重要信息，为社会学、政治学、经济学、传播学、心理学、法学、公共卫生和公共政策等学科的学者以及私营和政府部门的行为者提供信息。这一奖项反映了NSF的法定使命，并通过使用基金会的智力价值和更广泛的影响审查标准进行评估，被认为值得支持。

Vast amounts of survey data are collected each year for many purposes, including the collection of information for government use, public opinion and election surveys, advertising and marketing research, as well as basic research in the social and economic sciences. Because surveys are so widely used, it is essential that we establish a knowledge base about the measurement quality of survey data and the attributes of questions that produce the best data. Science well understands the reliability of multi-item measures, or scales, commonly used in such research. But we know much less about the quality of single-item survey questions. New data resources and modeling strategies now make it possible to assess the reliability or consistency of measurement for these survey measures. Based on statistical models for evaluating reliability of measurement, this project draws upon the insights from the information gathering process in surveys. The results of this research will provide practical suggestions that will improve the quality of survey data, thus improving survey research generally. The project will develop a publicly available website to allow all survey researchers to investigate question reliability of a large number of commonly used survey questions. Because surveys are used broadly by science, government and the private sector to inform many policy and economic decisions, improved information about survey measures will contribute to the health and well being of U. S. society across multiple sectors and over time.Because surveys are so widely used, it is essential that we establish a knowledge base about the measurement quality of survey data and the attributes of questions that produce the best results. In order to accomplish these aims, this project develops a set of meta-data containing information for roughly 1200 single-item questions representative of typical questions used in social science surveys. The database of survey questions, developed on the basis of ten nationally (or regionally) representative panel studies including the General Social Surveys, the Health and Retirement Study, the National Election Studies, contains estimates of question-specific reliabilities, along with detailed coding of attributes of the questions, such as their content, response formats, and question length, which can be used to evaluate the optimal properties of survey questions with respect to levels of measurement error. Through an analysis of the reliability information and the attributes of survey questions from these large-scale panel studies, the study focuses on three major elements of questionnaire construction: content, context and form. Statistical analysis will involve use of auto-regressive or Quasi-Markov simplex modeling techniques. The resulting meta-data set will be archived with the Inter-university Consortium for Political and Social Research (ICPSR) and other suitable data archives. In addition, the project will post key elements of these data on a publicly available website, along with images of questions and question context involved, in a manner that allows users to search, filter or query the data base in investigating the reliability of types of survey questions of interest. These findings will contribute important information to the multidisciplinary field of survey methodology, which informs scholars in disciplines including sociology, political science, economics, communication, psychology, law, public health, and public policy, as well as actors in the private and governmental sectors.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.