Developing and testing a computer tool that critiques survey questions

一项调查问题答案的有效性和可靠性，关键取决于受访者是否理解问题的含义。这个项目开发和测试了一种计算机工具，帮助调查设计者提高问题的理解性。计算机工具将有特定的模块来诊断调查中的每个问题，这些问题涉及不同水平的语言、话语和世界知识。例如，该评论指出了以下问题：低频词、含糊或含糊的术语、相关术语不明确、语法复杂、高工作记忆负荷、误导性预设以及似乎与调查背景无关的内容。该计算机工具将整合认知科学、人工智能、计算语言学、话语处理和心理学领域的经验发现和计算架构。其中一些模块是如此复杂、技术或微妙，以至于没有辅助的人眼是看不见的，包括调查方法、问卷设计和计算语言学的专家。这促使需要一种计算机工具，以协助研究方法学家修改问题和了解每个组成部分背后的复杂机制。计算机工具将是有用的，因为它为有问题的问题提供了准确和可靠的诊断。因此，该项目将从几个方面评价计算机工具的性能。每个模块确定特定问题是否有问题(例如，不熟悉的术语、工作内存过载)。这些决定将与专家的决定进行比较。还需要其他绩效衡量标准，因为训练有素的专家法官可能会错过微妙的计算机制。这些其他措施将评估计算机输出是否可以预测受访者回答问题时的行为：(A)受访者在对话面试中表示难以理解问题的行为(例如受访者的澄清问题)和(B)当受访者多次回答一个问题时，对问题答案的重测可靠性。还将比较原始问题、由不使用计算机工具的调查方法学家修订的问题和由使用该工具的调查方法学家修订的问题的绩效衡量。这项研究得到了方法学、测量和统计计划和联邦统计机构联盟的支持，该联盟隶属于调查方法学资助机会研究。

The validity and reliability of answers to questions on a survey critically depend on whether the respondents understand the meaning of the questions. This project develops and tests a computer tool that assists survey designers in improving the comprehensibility of questions. The computer tool will have particular modules that diagnose each question in a survey on various levels of language, discourse, and world knowledge. For example, the critique identifies questions with low frequency words, vague or ambiguous terms, unclear relative terms, complex syntax, high working memory load, misleading presuppositions, and content that appears to be unrelated to the survey context. The computer tool will incorporate empirical findings and computational architectures in the fields of cognitive science, artificial intelligence, computational linguistics, discourse processing, and psychology. Some of these modules are so complex, technical, or subtle that they are invisible to the unassisted human eye, including experts in survey methodology, questionnaire design, and computational linguistics. This motivates the need for a computer tool to assist the research methodologist in revising questions and in learning about the complex mechanisms that underlie each component.The computer tool will be useful to the extent that it provides an accurate and reliable diagnosis of problematic questions. The project will therefore evaluate the performance of the computer tool on several measures. Each module determines whether or not a particular question has a problem (e.g., unfamiliar technical term, working memory overload). These decisions will be compared with the decisions of experts. Other performance measures are needed because trained expert judges may miss subtle computational mechanisms. These other measures will assess whether the computer output can predict the behavior of respondents when they answer the questions: (a) behaviors of respondents that indicate they are having difficulty comprehending the question in a conversational interview (such as clarification questions of respondents) and (b) test-retest reliability of answers to questions when respondents answer a question on multiple occasions. Performance measures also will be compared for original questions, questions revised by survey methodologists who do not use the computer tool, and questions revised by survey methodologists who have had the benefit of using the tool. This research is supported by the Methodology, Measurement, and Statistics Program and a consortium of federal statistical agencies under the Research on Survey Methodology Funding Opportunity.