Developing and testing a computer tool that critiques survey questions

点击翻译按钮获取中文摘要

The validity and reliability of answers to questions on a survey critically depend on whether the respondents understand the meaning of the questions. This project develops and tests a computer tool that assists survey designers in improving the comprehensibility of questions. The computer tool will have particular modules that diagnose each question in a survey on various levels of language, discourse, and world knowledge. For example, the critique identifies questions with low frequency words, vague or ambiguous terms, unclear relative terms, complex syntax, high working memory load, misleading presuppositions, and content that appears to be unrelated to the survey context. The computer tool will incorporate empirical findings and computational architectures in the fields of cognitive science, artificial intelligence, computational linguistics, discourse processing, and psychology. Some of these modules are so complex, technical, or subtle that they are invisible to the unassisted human eye, including experts in survey methodology, questionnaire design, and computational linguistics. This motivates the need for a computer tool to assist the research methodologist in revising questions and in learning about the complex mechanisms that underlie each component.The computer tool will be useful to the extent that it provides an accurate and reliable diagnosis of problematic questions. The project will therefore evaluate the performance of the computer tool on several measures. Each module determines whether or not a particular question has a problem (e.g., unfamiliar technical term, working memory overload). These decisions will be compared with the decisions of experts. Other performance measures are needed because trained expert judges may miss subtle computational mechanisms. These other measures will assess whether the computer output can predict the behavior of respondents when they answer the questions: (a) behaviors of respondents that indicate they are having difficulty comprehending the question in a conversational interview (such as clarification questions of respondents) and (b) test-retest reliability of answers to questions when respondents answer a question on multiple occasions. Performance measures also will be compared for original questions, questions revised by survey methodologists who do not use the computer tool, and questions revised by survey methodologists who have had the benefit of using the tool. This research is supported by the Methodology, Measurement, and Statistics Program and a consortium of federal statistical agencies under the Research on Survey Methodology Funding Opportunity.