Developing and testing a computer tool that critiques survey questions

开发和测试批评调查问题的计算机工具

• 批准号: 9977969
• 负责人: Arthur Graesser
• 金额: $ 20.6万
• 依托单位: University of Memphis
• 依托单位国家: 美国
• 项目类别: Standard Grant
• 财政年份: 2000
• 资助国家: 美国
• 起止时间: 2000-01-01 至 2003-12-31
• 项目状态: 已结题
• 来源: https://www.nsf.gov/awardsearch/showAward?AWD_ID=9977969&HistoricalAwards=false
• 关键词: Developing; testing; computer; tool; critiques

The validity and reliability of answers to questions on a survey critically depend on whether the respondents understand the meaning of the questions. This project develops and tests a computer tool that assists survey designers in improving the comprehensibility of questions. The computer tool will have particular modules that diagnose each question in a survey on various levels of language, discourse, and world knowledge. For example, the critique identifies questions with low frequency words, vague or ambiguous terms, unclear relative terms, complex syntax, high working memory load, misleading presuppositions, and content that appears to be unrelated to the survey context. The computer tool will incorporate empirical findings and computational architectures in the fields of cognitive science, artificial intelligence, computational linguistics, discourse processing, and psychology. Some of these modules are so complex, technical, or subtle that they are invisible to the unassisted human eye, including experts in survey methodology, questionnaire design, and computational linguistics. This motivates the need for a computer tool to assist the research methodologist in revising questions and in learning about the complex mechanisms that underlie each component.The computer tool will be useful to the extent that it provides an accurate and reliable diagnosis of problematic questions. The project will therefore evaluate the performance of the computer tool on several measures. Each module determines whether or not a particular question has a problem (e.g., unfamiliar technical term, working memory overload). These decisions will be compared with the decisions of experts. Other performance measures are needed because trained expert judges may miss subtle computational mechanisms. These other measures will assess whether the computer output can predict the behavior of respondents when they answer the questions: (a) behaviors of respondents that indicate they are having difficulty comprehending the question in a conversational interview (such as clarification questions of respondents) and (b) test-retest reliability of answers to questions when respondents answer a question on multiple occasions. Performance measures also will be compared for original questions, questions revised by survey methodologists who do not use the computer tool, and questions revised by survey methodologists who have had the benefit of using the tool. This research is supported by the Methodology, Measurement, and Statistics Program and a consortium of federal statistical agencies under the Research on Survey Methodology Funding Opportunity.

调查中问题答案的有效性和可靠性主要取决于受访者是否理解问题的含义。 该项目开发和测试一种计算机工具，协助调查设计者提高问题的可理解性。 计算机工具将有特定的模块，诊断在语言，话语和世界知识的各个层次的调查中的每个问题。 例如，评论识别出的问题具有低频词、模糊或模棱两可的术语、不清楚的相关术语、复杂的句法、高工作记忆负荷、误导性的预设以及似乎与调查上下文无关的内容。 计算机工具将结合认知科学，人工智能，计算语言学，话语处理和心理学领域的经验发现和计算架构。 其中一些模块是如此复杂，技术性或微妙，它们是不可见的，以肉眼，包括专家在调查方法，问卷设计和计算语言学。 这就需要一种计算机工具来帮助研究方法学家修改问题和了解每个组成部分背后的复杂机制，计算机工具将是有用的，因为它提供了对有问题的问题的准确和可靠的诊断。 因此，该项目将从若干方面评价计算机工具的性能。 每个模块确定特定问题是否有问题（例如，不熟悉的技术术语，工作记忆过载）。 这些决定将与专家的决定进行比较。 需要其他性能指标，因为训练有素的专家法官可能会错过微妙的计算机制。 这些其他措施将评估计算机输出是否可以预测受访者回答问题时的行为：（a）受访者的行为表明他们在对话式采访中难以理解问题（例如受访者的澄清问题）和（B）当受访者多次回答问题时，问题答案的重测信度。 绩效指标还将比较原始问题，调查方法学家谁不使用计算机工具修改的问题，和调查方法学家谁有使用该工具的好处修改的问题。这项研究得到了方法、测量和统计计划以及联邦统计机构联盟的支持，该联盟隶属于调查方法资助机会研究。