Developing and testing a computer tool that critiques survey questions

开发和测试批评调查问题的计算机工具

• 批准号: 9977969
• 负责人: Arthur Graesser
• 金额: $ 20.6万
• 依托单位: University of Memphis
• 依托单位国家: 美国
• 项目类别: Standard Grant
• 财政年份: 2000
• 资助国家: 美国
• 起止时间: 2000-01-01 至 2003-12-31
• 项目状态: 已结题
• 来源: https://www.nsf.gov/awardsearch/showAward?AWD_ID=9977969&HistoricalAwards=false
• 关键词: Developing; testing; computer; tool; critiques

The validity and reliability of answers to questions on a survey critically depend on whether the respondents understand the meaning of the questions. This project develops and tests a computer tool that assists survey designers in improving the comprehensibility of questions. The computer tool will have particular modules that diagnose each question in a survey on various levels of language, discourse, and world knowledge. For example, the critique identifies questions with low frequency words, vague or ambiguous terms, unclear relative terms, complex syntax, high working memory load, misleading presuppositions, and content that appears to be unrelated to the survey context. The computer tool will incorporate empirical findings and computational architectures in the fields of cognitive science, artificial intelligence, computational linguistics, discourse processing, and psychology. Some of these modules are so complex, technical, or subtle that they are invisible to the unassisted human eye, including experts in survey methodology, questionnaire design, and computational linguistics. This motivates the need for a computer tool to assist the research methodologist in revising questions and in learning about the complex mechanisms that underlie each component.The computer tool will be useful to the extent that it provides an accurate and reliable diagnosis of problematic questions. The project will therefore evaluate the performance of the computer tool on several measures. Each module determines whether or not a particular question has a problem (e.g., unfamiliar technical term, working memory overload). These decisions will be compared with the decisions of experts. Other performance measures are needed because trained expert judges may miss subtle computational mechanisms. These other measures will assess whether the computer output can predict the behavior of respondents when they answer the questions: (a) behaviors of respondents that indicate they are having difficulty comprehending the question in a conversational interview (such as clarification questions of respondents) and (b) test-retest reliability of answers to questions when respondents answer a question on multiple occasions. Performance measures also will be compared for original questions, questions revised by survey methodologists who do not use the computer tool, and questions revised by survey methodologists who have had the benefit of using the tool. This research is supported by the Methodology, Measurement, and Statistics Program and a consortium of federal statistical agencies under the Research on Survey Methodology Funding Opportunity.

对调查问题答案的有效性和可靠性在很大程度上取决于受访者是否理解问题的含义。 该项目开发和测试了一种计算机工具，该工具可帮助调查设计师改善问题的可理解性。 计算机工具将具有特定的模块，可以在各种语言，话语和世界知识的调查中诊断每个问题。 例如，批评以低频单词，模糊或模棱两可的术语，不清楚的相对术语，复杂的语法，高工作记忆负载，误导性的预设以及似乎与调查环境无关的内容来确定问题。 该计算机工具将在认知科学，人工智能，计算语言学，话语处理和心理学领域中结合经验发现和计算体系结构。 这些模块中的一些是如此复杂，技术或微妙，以至于无助的人眼看不见，包括调查方法，问卷设计和计算语言学专家。 这激发了计算机工具的需求，以帮助研究方法学家修改问题，并了解每个组件的复杂机制。计算机工具将在提供有问题问题的准确可靠诊断的范围内很有用。 因此，该项目将在多个措施上评估计算机工具的性能。 每个模块确定一个特定问题是否有问题（例如，陌生的技术术语，工作记忆超载）。 这些决定将与专家的决定进行比较。 需要其他绩效指标，因为训练有素的专家法官可能会错过微妙的计算机制。 这些其他措施将评估计算机输出在回答问题时是否可以预测受访者的行为：（a）受访者的行为表明他们在对话访谈中很难理解问题（例如，受访者的澄清问题）和（b）当受访者在多个场合回答问题时，对问题的回答可靠性。 还将比较原始问题的绩效指标，不使用计算机工具的调查方法学家修订的问题以及受益于使用该工具的调查方法学家修订的问题。这项研究得到了根据调查方法资助机会研究的方法，测量和统计计划以及联邦统计机构的联盟的支持。