The Reliability of Survey Data

这是对调查数据可靠性的方法论研究。鉴于社会、经济和行为科学严重依赖调查结果和其他收集重要数据的问答方法，了解调查结果的总体可靠性或可靠性至关重要。该项目将对调查问题答复的可靠性进行系统分析，以更好地了解调查数据的性质和产生这些数据的过程。第二个目标是就调查问题的属性提出切实可行的建议，以最大限度地提高其测量的可靠性。分析的单位是调查问题。该项目将建立一个调查问题数据库，其中包括6次对美国全国人口进行的大规模小组调查中的1,000多个变量的可靠性问题级信息和问题特征，这将为检验有关调查测量可靠性变化的几个假设提供基础。调查访谈被认为是一种收集信息的方法，它涉及许多关键因素--理解、可访问性、检索和沟通--这些因素对数据质量起到了影响作用。从理论上讲，这些因素造成测量误差的方式可以与调查问题和测量环境联系在一起。与其他方法不同的是，这项研究将估计单一调查问题的可靠性，并将检查可靠性作为分析方法差异的函数的变化。具体地说，该项目将比较基于结构方程模型的可靠性估计和基于项目反应理论的Rasch模型的估计，并将探索针对同一组问题的类别数据方法。所采取的方法依赖于多波面板数据来估计可靠性。将使用来自美国概率样本的六个这样的小组研究：1)1956-58-60全国选举研究(NES)小组，2)1972-74-76 NES小组，3)1990-91-92 NES小组，4)1992-93-94 NES小组，5)1986-94美国人改变生活(ACL)小组研究，6)美国家庭研究(SAF)1980-85-93关于年轻人的三波小组及其母亲超过5波，始于1961年。这项研究将解决以下问题：a)调查数据总体上有多可靠？B)测量的可靠性是否取决于被测量内容的性质？特别是，事实信息的收集是否比态度和/或其他主观数据更准确？而且，非事实问题的类型(态度、信念和自我评估)在可靠性上是否有所不同？C)测量的可靠性是否随测量方法的特性而变化？在事实数据的情况下，代理报告是否与自我报告一样可靠？面试官的观察结果有多可靠？D)问题的性质是否影响数据质量？例如，在态度测量中，问题措辞的性质、回答类别的数量、言语标记的程度以及回答格式的其他属性对可靠性有何影响？E)关于态度和其他主观数据，改变问题的背景是否会影响可靠性？这项研究建议在一系列工作假设的框架内评估这些领域的问题，这些假设源于关于测量误差来源的理论和研究经验。简而言之，分析将集中于解释由于这几个因素而导致的可靠性变化。从小组调查中收集有关测量可靠性的信息，不仅将提高对特定形式的调查测量的优缺点的了解，还将为开发关于调查测量可靠性的大规模数据库奠定基础，该数据库可以解决社会、经济和行为科学中数据质量的基本问题。增加关于数据质量这些方面的知识将有助于评估收集调查数据的技术，并将提高调查测量的质量。

This is a methodological study of the reliability of survey data. Given the heavy reliance of the social, economic and behavioral sciences on the results of surveys and other question-answer methods of gathering important data, it is essential to know the overall reliability or dependability of survey results. This project will conduct a systematic analysis of the measurement of reliability of responses to survey questions, to better understand the nature of survey data and the processes by which they are generated. A secondary goal is to make practical recommendations about the attributes of survey questions that will maximize their reliability of measurement. The unit of analysis is the survey question. The project will assemble a database for survey questions, consisting of question-level information on reliability and question characteristics for more than 1,000 variables from six large-scale panel surveys of national populations in the U.S., which will provide a basis for testing several hypotheses regarding variation in reliability of survey measurement. The survey interview is conceived of as a method for gathering information, which involves a number of critical elements --comprehension, accessibility, retrieval, and communication -- that play a role in affecting the quality of data. These factors are theorized to contribute to measurement errors in a way that can be linked to the survey question and the context of measurement. In contrast to other approaches, this research will estimate the reliability of single survey questions, and will examine variations in reliability as a function of differences in the method of analysis. Specifically, the project will compare estimates of reliability based on structural equation models with those based on Rasch models based on item-response theory, and will explore categoric data approaches to the same set of issues. The approach taken relies on multi-wave panel data for the estimation of reliability. Six such panel studies from probabi lity samples of the U.S. will be used: 1) the 1956-58-60 National Election Study (NES) panel, 2) the 1972-74-76 NES panel, 3) the 1990-91-92 NES panel, 4) the 1992-93-94 NES panel, 5) the 1986-94 Americans' Changing Lives (ACL) panel study, and 6) the Study of American Families (SAF) 1980-85-93 three-wave panel on young adults and more than 5 waves on their mothers, dating to 1961. The research will address the following question areas: a) How reliable are survey data in general? b) Does reliability of measurement depend on the nature of the content being measured? In particular, is factual information gathered more precisely than attitudinal and/or other subjective data? And, do types of nonfactual questions (attitudes, beliefs and self-assessments) differ in reliability? c) Does reliability of measurement vary as a function of the properties of the approach to measurement? In the case of factual data, are proxy reports as reliable as self-reports? How reliable are interviewer observations? d) Does the nature of the question affect the quality of data? For example, in attitude measurement, how is reliability affected by the nature of the question-phrasing, the number of response categories, the extent of verbal labeling, and other properties of the response format? e) With respect to attitude and other subjective data, is reliability affected by changing the context of the questions? The research proposes to assess questions in these areas within the framework of a set of working hypotheses derived from theory and research experience on the sources of measurement errors. Simply put, the analysis will focus on explaining variation in reliability due to these several factors. Assembling information on measurement reliability from the panel surveys will not only improve knowledge about the strengths and weaknesses of particular forms of survey measurement but will also lay the groundwork for developing a large-scale data base on survey measurement reliability that can addr ess basic issues of data quality in the social, economic and behavioral sciences. Increased knowledge regarding these aspects of data quality will help evaluate techniques for gathering survey data and will improve the quality of survey measurement.