The Reliability of Survey Data

这是对调查数据可靠性的方法学研究。鉴于社会、经济和行为科学对调查结果和其他收集重要数据的问答方法的严重依赖，了解调查结果的总体可靠性或可靠性至关重要。这个项目将对调查问题的回答可靠性的测量进行系统的分析，以便更好地了解调查数据的性质和产生这些数据的过程。第二个目标是对调查问题的属性提出实用的建议，以最大限度地提高测量的可靠性。分析的单位是调查问题。该项目将为调查问题建立一个数据库，其中包括从美国全国人口的六次大规模小组调查中获得的关于可靠性和1000多个变量的问题级别信息和问题特征，这将为检验关于调查测量可靠性变化的若干假设提供基础。调查访谈被认为是一种收集信息的方法，它涉及许多关键因素——理解、可访问性、检索和交流——这些因素在影响数据质量方面起着重要作用。从理论上讲，这些因素会导致测量误差，这种误差可以与调查问题和测量背景联系起来。与其他方法相比，本研究将估计单个调查问题的可靠性，并将检查可靠性的变化作为分析方法差异的函数。具体来说，该项目将比较基于结构方程模型的可靠性估计与基于项目反应理论的Rasch模型的可靠性估计，并将探索针对同一组问题的分类数据方法。所采用的方法依赖于多波面板数据进行可靠性估计。本文将使用来自美国概率样本的六个这样的小组研究：1)1956-58-60年全国选举研究（NES）小组，2)1972-74-76年全国选举研究小组，3)1990-91-92年全国选举研究小组，4)1992-93-94年全国选举研究小组，5)1986-94年美国人改变生活（ACL）小组研究，6)美国家庭研究（SAF） 1980-85-93年年轻人三波小组和他们的母亲超过5波小组，从1961年开始。该研究将解决以下问题领域：a)总体上调查数据的可靠性如何？b)测量的可靠性是否取决于被测量内容的性质？特别是，事实信息的收集是否比态度和/或其他主观数据更精确？非事实性问题（态度、信念和自我评估）的类型在可靠性上是否有所不同？c)测量的可靠性是否随测量方法的特性而变化？在事实数据的情况下，代理报告和自我报告一样可靠吗？面试官的观察结果有多可靠？d)问题的性质是否影响数据的质量？例如，在态度测量中，可靠性如何受到问题措辞的性质、回答类别的数量、言语标记的程度和回答格式的其他属性的影响？e)对于态度和其他主观数据，改变问题的背景是否会影响信度？该研究建议在一套基于测量误差来源的理论和研究经验的工作假设的框架内评估这些领域的问题。简单地说，分析将侧重于解释由于这几个因素导致的可靠性变化。从小组调查中收集关于测量可靠性的信息不仅可以提高对特定形式的调查测量的优缺点的认识，而且还可以为开发一个关于调查测量可靠性的大规模数据库奠定基础，该数据库可以处理社会、经济和行为科学中数据质量的基本问题。增加关于数据质量这些方面的知识将有助于评价收集调查数据的技术，并将提高调查测量的质量。

This is a methodological study of the reliability of survey data. Given the heavy reliance of the social, economic and behavioral sciences on the results of surveys and other question-answer methods of gathering important data, it is essential to know the overall reliability or dependability of survey results. This project will conduct a systematic analysis of the measurement of reliability of responses to survey questions, to better understand the nature of survey data and the processes by which they are generated. A secondary goal is to make practical recommendations about the attributes of survey questions that will maximize their reliability of measurement. The unit of analysis is the survey question. The project will assemble a database for survey questions, consisting of question-level information on reliability and question characteristics for more than 1,000 variables from six large-scale panel surveys of national populations in the U.S., which will provide a basis for testing several hypotheses regarding variation in reliability of survey measurement. The survey interview is conceived of as a method for gathering information, which involves a number of critical elements --comprehension, accessibility, retrieval, and communication -- that play a role in affecting the quality of data. These factors are theorized to contribute to measurement errors in a way that can be linked to the survey question and the context of measurement. In contrast to other approaches, this research will estimate the reliability of single survey questions, and will examine variations in reliability as a function of differences in the method of analysis. Specifically, the project will compare estimates of reliability based on structural equation models with those based on Rasch models based on item-response theory, and will explore categoric data approaches to the same set of issues. The approach taken relies on multi-wave panel data for the estimation of reliability. Six such panel studies from probabi lity samples of the U.S. will be used: 1) the 1956-58-60 National Election Study (NES) panel, 2) the 1972-74-76 NES panel, 3) the 1990-91-92 NES panel, 4) the 1992-93-94 NES panel, 5) the 1986-94 Americans' Changing Lives (ACL) panel study, and 6) the Study of American Families (SAF) 1980-85-93 three-wave panel on young adults and more than 5 waves on their mothers, dating to 1961. The research will address the following question areas: a) How reliable are survey data in general? b) Does reliability of measurement depend on the nature of the content being measured? In particular, is factual information gathered more precisely than attitudinal and/or other subjective data? And, do types of nonfactual questions (attitudes, beliefs and self-assessments) differ in reliability? c) Does reliability of measurement vary as a function of the properties of the approach to measurement? In the case of factual data, are proxy reports as reliable as self-reports? How reliable are interviewer observations? d) Does the nature of the question affect the quality of data? For example, in attitude measurement, how is reliability affected by the nature of the question-phrasing, the number of response categories, the extent of verbal labeling, and other properties of the response format? e) With respect to attitude and other subjective data, is reliability affected by changing the context of the questions? The research proposes to assess questions in these areas within the framework of a set of working hypotheses derived from theory and research experience on the sources of measurement errors. Simply put, the analysis will focus on explaining variation in reliability due to these several factors. Assembling information on measurement reliability from the panel surveys will not only improve knowledge about the strengths and weaknesses of particular forms of survey measurement but will also lay the groundwork for developing a large-scale data base on survey measurement reliability that can addr ess basic issues of data quality in the social, economic and behavioral sciences. Increased knowledge regarding these aspects of data quality will help evaluate techniques for gathering survey data and will improve the quality of survey measurement.