Statistical Quality Control of Low-Stakes Assessment Data

这个研究项目将发展统计方法来监测和控制低风险评估和问卷数据的质量。社会和行为科学领域的研究人员提供的许多评估测试和调查被参与者认为是低风险的，然而研究人员严重依赖这些数据来解决他们的研究问题。来自低风险评估的数据可能包含很大一部分注意力不集中的回答，这是由参与者的粗心或疲劳造成的，有时是由调查机器人的恶意企图造成的。参与者招募和数据收集在线平台的日益普及进一步加剧了这一问题。研究人员将开发统计方法来检测注意力不集中的反应。他们将对这些方法的性能进行基准测试，并确定针对不同类型注意力不集中的最佳方法。作为项目的一部分，将为研究人员创建指南、协议和用户友好的软件。参与该项目的学生将获得统计质量控制方面的培训和研究经验。要开发的方法和工具将影响多个学科，包括心理学、教育、市场营销和公共卫生。该项目将侧重于在多维评估和使用多义条目的问卷中发现注意力不集中的反应。多同义词是低风险语境中使用最广泛的一种词。统计方法将基于多维分级反应模型，一种项目反应理论（IRT）模型，可以捕获多维多分反应数据。需要开发的方法包括基于irt模型的人拟合统计和最快速的变化检测方法，如变化点分析和累积和控制图。当数据中存在许多异常的响应模式时，异常情况就很难被发现，这种现象被称为“掩蔽效应”。为了抵消这种影响，这些方法的健壮版本将通过迭代地降低外围情况的权重来开发。这些方法的性能将通过广泛的蒙特卡罗模拟来评估，模拟各种现实生活场景。该奖项反映了美国国家科学基金会的法定使命，并通过使用基金会的知识价值和更广泛的影响审查标准进行评估，被认为值得支持。

This research project will develop statistical methods to monitor and control the quality of low-stakes assessment and questionnaire data. Many assessment tests and surveys given by researchers in the social and behavioral sciences are perceived as low stakes by participants, yet researchers rely heavily on such data to address their research questions. Data from low-stakes assessments may contain a substantial portion of inattentive responses, driven by carelessness or fatigue from participants, or sometimes from malicious attempts by survey-bots. The increasing popularity of online platforms for participant recruitment and data collection further exacerbates the problem. The investigators will develop statistical methods to detect inattentive responses. They will benchmark the performance of these methods and identify the best method for different types of inattentiveness. Guidelines, protocols, and user-friendly software for researchers will be created as part of the project. Students participating in this project will receive training and research experience in statistical quality control. The methods and tools to be developed will impact multiple disciplines, including psychology, education, marketing, and public health.This project will focus on the detection of inattentive responses in multidimensional assessments and questionnaires that use polytomous items. Polytomous items are the most widely used type of items in low-stakes contexts. Statistical methods will be developed based on the multidimensional graded response model, a type of item response theory (IRT) model that can capture multidimensional polytomous response data. Methods to be developed include IRT-model-based person-fit statistics and quickest change detection methods, such as change point analysis and cumulative sum control chart. When many aberrant response patterns exist in the data, outlying cases become difficult to detect, a phenomenon known as the "masking effect." To counteract this effect, robust versions of these methods will be developed by iteratively down-weighting outlying cases. The performance of these methods will be evaluated through extensive Monte Carlo simulations that mimic various real-life scenarios.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.