Statistical Quality Control of Low-Stakes Assessment Data

低风险评估数据的统计质量控制

• 批准号: 1853166
• 负责人: Ying Cheng
• 金额: $ 33万
• 依托单位: University of Notre Dame
• 依托单位国家: 美国
• 项目类别: Standard Grant
• 财政年份: 2019
• 资助国家: 美国
• 起止时间: 2019-09-01 至 2024-08-31
• 项目状态: 已结题
• 来源: https://www.nsf.gov/awardsearch/showAward?AWD_ID=1853166&HistoricalAwards=false
• 关键词: Statistical; Quality; Control; Low; Stakes

This research project will develop statistical methods to monitor and control the quality of low-stakes assessment and questionnaire data. Many assessment tests and surveys given by researchers in the social and behavioral sciences are perceived as low stakes by participants, yet researchers rely heavily on such data to address their research questions. Data from low-stakes assessments may contain a substantial portion of inattentive responses, driven by carelessness or fatigue from participants, or sometimes from malicious attempts by survey-bots. The increasing popularity of online platforms for participant recruitment and data collection further exacerbates the problem. The investigators will develop statistical methods to detect inattentive responses. They will benchmark the performance of these methods and identify the best method for different types of inattentiveness. Guidelines, protocols, and user-friendly software for researchers will be created as part of the project. Students participating in this project will receive training and research experience in statistical quality control. The methods and tools to be developed will impact multiple disciplines, including psychology, education, marketing, and public health.This project will focus on the detection of inattentive responses in multidimensional assessments and questionnaires that use polytomous items. Polytomous items are the most widely used type of items in low-stakes contexts. Statistical methods will be developed based on the multidimensional graded response model, a type of item response theory (IRT) model that can capture multidimensional polytomous response data. Methods to be developed include IRT-model-based person-fit statistics and quickest change detection methods, such as change point analysis and cumulative sum control chart. When many aberrant response patterns exist in the data, outlying cases become difficult to detect, a phenomenon known as the "masking effect." To counteract this effect, robust versions of these methods will be developed by iteratively down-weighting outlying cases. The performance of these methods will be evaluated through extensive Monte Carlo simulations that mimic various real-life scenarios.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.

这个研究项目将发展统计方法来监测和控制低风险评估和问卷数据的质量。社会和行为科学领域的研究人员提供的许多评估测试和调查被参与者认为是低风险的，然而研究人员严重依赖这些数据来解决他们的研究问题。来自低风险评估的数据可能包含很大一部分注意力不集中的回答，这是由参与者的粗心或疲劳造成的，有时是由调查机器人的恶意企图造成的。参与者招募和数据收集在线平台的日益普及进一步加剧了这一问题。研究人员将开发统计方法来检测注意力不集中的反应。他们将对这些方法的性能进行基准测试，并确定针对不同类型注意力不集中的最佳方法。作为项目的一部分，将为研究人员创建指南、协议和用户友好的软件。参与该项目的学生将获得统计质量控制方面的培训和研究经验。要开发的方法和工具将影响多个学科，包括心理学、教育、市场营销和公共卫生。该项目将侧重于在多维评估和使用多义条目的问卷中发现注意力不集中的反应。多同义词是低风险语境中使用最广泛的一种词。统计方法将基于多维分级反应模型，一种项目反应理论（IRT）模型，可以捕获多维多分反应数据。需要开发的方法包括基于irt模型的人拟合统计和最快速的变化检测方法，如变化点分析和累积和控制图。当数据中存在许多异常的响应模式时，异常情况就很难被发现，这种现象被称为“掩蔽效应”。为了抵消这种影响，这些方法的健壮版本将通过迭代地降低外围情况的权重来开发。这些方法的性能将通过广泛的蒙特卡罗模拟来评估，模拟各种现实生活场景。该奖项反映了美国国家科学基金会的法定使命，并通过使用基金会的知识价值和更广泛的影响审查标准进行评估，被认为值得支持。

项目成果

Application of Change Point Analysis of Response Time Data to Detect Test Speededness

应用响应时间数据变点分析来检测测试速度

• DOI: 10.1177/00131644211046392
• 发表时间: 2022
• 期刊: Educational and psychological measurement
• 影响因子: 2.7
• 作者: Ying Cheng;Can Shao
• 通讯作者: Can Shao

Asymptotically Corrected Person Fit Statistics for Multidimensional Constructs with Simple Structure and Mixed Item Types

具有简单结构和混合项目类型的多维结构的渐近校正人体拟合统计

• DOI: 10.1007/s11336-021-09756-3
• 发表时间: 2021
• 期刊: Psychometrika
• 影响因子: 3
• 作者: Hong, Maxwell;Lin, Lizhen;Cheng, Ying
• 通讯作者: Cheng, Ying

Robust Estimation for Response Time Modeling

响应时间建模的鲁棒估计

• DOI: 10.1111/jedm.12286
• 发表时间: 2021
• 期刊: Journal of Educational Measurement
• 影响因子: 1.3
• 作者: Hong, Maxwell;Rebouças, Daniella A.;Cheng, Ying
• 通讯作者: Cheng, Ying

Data Exclusion in Policy Survey and Questionnaire Data: Aberrant Responses and Missingness

政策调查和问卷数据中的数据排除：异常反应和缺失

• DOI: 10.1177/23727322221144650
• 发表时间: 2023
• 期刊: Policy Insights from the Behavioral and Brain Sciences
• 影响因子: 3.8
• 作者: Hong, Maxwell;Carter, Matthew;Kim, Casey;Cheng, Ying
• 通讯作者: Cheng, Ying

A Comprehensive Review and Comparison of CUSUM and Change-Point-Analysis Methods to Detect Test Speededness

检测测试速度的 CUSUM 和变点分析方法的全面回顾和比较

• DOI: 10.1080/00273171.2020.1809981
• 发表时间: 2021
• 期刊: Multivariate Behavioral Research
• 影响因子: 3.8
• 作者: Yu, Xiaofeng;Cheng, Ying
• 通讯作者: Cheng, Ying