Statistical Quality Control of Low-Stakes Assessment Data

本研究项目将开发统计方法，以监测和控制低风险评估和问卷数据的质量。社会和行为科学研究人员提供的许多评估测试和调查被参与者视为低风险，但研究人员严重依赖这些数据来解决他们的研究问题。来自低风险评估的数据可能包含很大一部分不注意的反应，这是由参与者的疏忽或疲劳造成的，有时也是由调查机器人的恶意尝试造成的。招募参与者和收集数据的在线平台日益普及，进一步加剧了这一问题。研究人员将开发统计方法来检测疏忽的反应。他们将对这些方法的性能进行基准测试，并确定针对不同类型的疏忽的最佳方法。作为该项目的一部分，将为研究人员制定指导方针、协议和用户友好型软件。参加该项目的学生将获得统计质量控制方面的培训和研究经验。开发的方法和工具将影响多个学科，包括心理学，教育，市场营销和公共卫生。该项目将侧重于在多维度评估和使用多分类项目的问卷调查中检测注意力不集中的反应。多分类项目是低风险背景下使用最广泛的项目类型。将基于多维分级反应模型开发统计方法，这是一种可以捕获多维多分类反应数据的项目反应理论（IRT）模型。待开发的方法包括基于IRT模型的个人拟合统计和最快变化检测方法，如变点分析和累积和控制图。当数据中存在许多异常反应模式时，异常情况就很难检测出来，这种现象被称为“掩蔽效应”。“为了抵消这种影响，将通过迭代地降低离群情况的权重来开发这些方法的鲁棒版本。这些方法的性能将通过广泛的蒙特卡罗模拟来评估，模拟各种现实生活中的场景。这个奖项反映了NSF的法定使命，并已被认为是值得通过使用基金会的智力价值和更广泛的影响审查标准进行评估的支持。

This research project will develop statistical methods to monitor and control the quality of low-stakes assessment and questionnaire data. Many assessment tests and surveys given by researchers in the social and behavioral sciences are perceived as low stakes by participants, yet researchers rely heavily on such data to address their research questions. Data from low-stakes assessments may contain a substantial portion of inattentive responses, driven by carelessness or fatigue from participants, or sometimes from malicious attempts by survey-bots. The increasing popularity of online platforms for participant recruitment and data collection further exacerbates the problem. The investigators will develop statistical methods to detect inattentive responses. They will benchmark the performance of these methods and identify the best method for different types of inattentiveness. Guidelines, protocols, and user-friendly software for researchers will be created as part of the project. Students participating in this project will receive training and research experience in statistical quality control. The methods and tools to be developed will impact multiple disciplines, including psychology, education, marketing, and public health.This project will focus on the detection of inattentive responses in multidimensional assessments and questionnaires that use polytomous items. Polytomous items are the most widely used type of items in low-stakes contexts. Statistical methods will be developed based on the multidimensional graded response model, a type of item response theory (IRT) model that can capture multidimensional polytomous response data. Methods to be developed include IRT-model-based person-fit statistics and quickest change detection methods, such as change point analysis and cumulative sum control chart. When many aberrant response patterns exist in the data, outlying cases become difficult to detect, a phenomenon known as the "masking effect." To counteract this effect, robust versions of these methods will be developed by iteratively down-weighting outlying cases. The performance of these methods will be evaluated through extensive Monte Carlo simulations that mimic various real-life scenarios.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.