The Reliability of Survey Data

这是对调查数据可靠性的方法研究。 鉴于社会、经济和行为科学对调查结果和其他收集重要数据的问答方法的严重依赖，了解调查结果的总体可靠性或可靠性至关重要。该项目将对调查问题回答的可靠性测量进行系统分析，以更好地了解调查数据的性质及其生成过程。第二个目标是就调查问题的属性提出切实可行的建议，以最大限度地提高测量的可靠性。 分析的单位是调查问题。该项目将为调查问题建立一个数据库，其中包括来自美国全国人口六次大规模小组调查的1,000多个变量的可靠性和问题特征的问题级信息，这将为检验关于调查测量可靠性变化的几个假设提供基础。调查访谈被认为是一种收集信息的方法，其中涉及一些关键要素-理解、可获得性、检索和交流-这些要素在影响数据质量方面发挥作用。这些因素被理论化，以某种方式促成测量误差，这种方式可以与调查问题和测量背景相联系。与其他方法相比，本研究将估计单个调查问题的可靠性，并将研究可靠性的变化作为分析方法差异的函数。具体而言，该项目将比较基于结构方程模型的可靠性估计与基于项目反应理论的Rasch模型的可靠性估计，并将探索同一组问题的分类数据方法。 所采取的方法依赖于多波面板数据的可靠性估计。将使用来自美国概率样本的六项此类小组研究：1）1956-58-60年全国选举研究（内斯）小组，2）1972-74-76年内斯小组，3）1990-91-92年内斯小组，4）1992-93-94年内斯小组，5）1986-94年美国人改变生活（ACL）小组研究，6）美国家庭研究（SAF）1980-85-93三波小组对年轻人和超过5波对他们的母亲，追溯到1961年。 研究将涉及以下问题领域：a）调查数据一般有多可靠？ B）测量的可靠性是否取决于被测量内容的性质？ 特别是，收集的事实信息是否比态度和/或其他主观数据更准确？ 而且，不同类型的非事实性问题（态度、信念和自我评估）的可靠性是否不同？ c）测量的可靠性是否随测量方法的特性而变化？ 在事实数据的情况下，代理报告和自我报告一样可靠吗？ 面试官的观察有多可靠？ （d）问题的性质是否影响数据的质量？ 例如，在态度测量中，问题措辞的性质、回答类别的数量、言语标记的程度以及回答格式的其他属性如何影响可靠性？ （e）关于态度和其他主观数据，可靠性是否会因改变问题的背景而受到影响？ 本研究建议在一套工作假设的框架内评估这些领域的问题，这些假设来自测量误差来源的理论和研究经验。简单地说，分析将集中在解释由于这几个因素而导致的可靠性变化。 从小组调查中收集关于计量可靠性的信息不仅将增进对特定形式的调查计量的优缺点的了解，而且还将为建立一个关于调查计量可靠性的大规模数据库奠定基础，该数据库可以解决社会、经济和行为科学中的数据质量基本问题。增加对数据质量这些方面的了解将有助于评价收集调查数据的技术，并将提高调查计量的质量。

This is a methodological study of the reliability of survey data. Given the heavy reliance of the social, economic and behavioral sciences on the results of surveys and other question-answer methods of gathering important data, it is essential to know the overall reliability or dependability of survey results. This project will conduct a systematic analysis of the measurement of reliability of responses to survey questions, to better understand the nature of survey data and the processes by which they are generated. A secondary goal is to make practical recommendations about the attributes of survey questions that will maximize their reliability of measurement. The unit of analysis is the survey question. The project will assemble a database for survey questions, consisting of question-level information on reliability and question characteristics for more than 1,000 variables from six large-scale panel surveys of national populations in the U.S., which will provide a basis for testing several hypotheses regarding variation in reliability of survey measurement. The survey interview is conceived of as a method for gathering information, which involves a number of critical elements --comprehension, accessibility, retrieval, and communication -- that play a role in affecting the quality of data. These factors are theorized to contribute to measurement errors in a way that can be linked to the survey question and the context of measurement. In contrast to other approaches, this research will estimate the reliability of single survey questions, and will examine variations in reliability as a function of differences in the method of analysis. Specifically, the project will compare estimates of reliability based on structural equation models with those based on Rasch models based on item-response theory, and will explore categoric data approaches to the same set of issues. The approach taken relies on multi-wave panel data for the estimation of reliability. Six such panel studies from probabi lity samples of the U.S. will be used: 1) the 1956-58-60 National Election Study (NES) panel, 2) the 1972-74-76 NES panel, 3) the 1990-91-92 NES panel, 4) the 1992-93-94 NES panel, 5) the 1986-94 Americans' Changing Lives (ACL) panel study, and 6) the Study of American Families (SAF) 1980-85-93 three-wave panel on young adults and more than 5 waves on their mothers, dating to 1961. The research will address the following question areas: a) How reliable are survey data in general? b) Does reliability of measurement depend on the nature of the content being measured? In particular, is factual information gathered more precisely than attitudinal and/or other subjective data? And, do types of nonfactual questions (attitudes, beliefs and self-assessments) differ in reliability? c) Does reliability of measurement vary as a function of the properties of the approach to measurement? In the case of factual data, are proxy reports as reliable as self-reports? How reliable are interviewer observations? d) Does the nature of the question affect the quality of data? For example, in attitude measurement, how is reliability affected by the nature of the question-phrasing, the number of response categories, the extent of verbal labeling, and other properties of the response format? e) With respect to attitude and other subjective data, is reliability affected by changing the context of the questions? The research proposes to assess questions in these areas within the framework of a set of working hypotheses derived from theory and research experience on the sources of measurement errors. Simply put, the analysis will focus on explaining variation in reliability due to these several factors. Assembling information on measurement reliability from the panel surveys will not only improve knowledge about the strengths and weaknesses of particular forms of survey measurement but will also lay the groundwork for developing a large-scale data base on survey measurement reliability that can addr ess basic issues of data quality in the social, economic and behavioral sciences. Increased knowledge regarding these aspects of data quality will help evaluate techniques for gathering survey data and will improve the quality of survey measurement.