Using response time to investigate students' test-taking behaviors in a NAEP computer-based study
Using response time to investigate students' test-taking behaviors in a NAEP computer-based study
复制标题
DOI：
10.1186/s40536-014-0008-1
复制
发表时间：
2014-12-01
影响因子：
3.1
通讯作者：
Jia, Yue
中科院分区：
文献类型：
--
作者：
Lee, Yi-Hsuan;Jia, Yue
背景：几十年来，大规模调查评估一直用于监测学生的知识和能力。这种评估旨在为不同人群提供群体水平的分数，对个别学生的考试成绩影响很小或没有影响。调查评估中学生的应试行为，特别是应试努力程度及其对成绩的影响一直是一个长期存在的问题。本文提出了一种程序来检查考试行为，使用从国家教育进步评估（NAEP）计算机为基础的研究中收集的反应时间，称为MCBS。方法：提出了一种更系统地识别快速猜测行为的五步程序。它涉及一种非基于模型的方法，该方法将学生项目对分类为反映解决方案行为或快速猜测行为。在验证步骤中加入了三个效度检查，以确保进一步调查前时间界限的合理性。行为分类的结果通过三种方法进行总结，探讨学生的应试行为是否与学生特征、项目特征或两者相关，以及如何相关。结果：在MCBS中，效度检验提供了令人信服的证据，证明推荐的阈值识别方法在区分快速猜测行为和解决行为方面是有效的。与现有的不同评估结果相比，快速猜测行为的比例非常低。对于该数据集，快速猜测行为对IRT建模中参数估计的影响最小。然而，当学生收到与他们的表现水平不匹配的项目时，他们明显表现出不同的行为。我们还发现学生的反应时间努力和自我报告之间存在差异，但根据观察到的数据，尚不清楚这种差异是否与学生如何解释背景问题有关。结论：本文提供了一种方法来解决识别快速猜测行为的问题，并阐明了学生参与NAEP的程度及其影响的问题，而不依赖于学生的自我评估或额外的测试设计成本。它揭示了NAEP评估设置中有关应试行为的有用信息，这些信息在文献中是不可用的。该程序适用于未来的标准NAEP评估，以及可获得时间数据的其他测试。
Background: Large-scale survey assessments have been used for decades to monitor what students know and can do. Such assessments aim at providing group-level scores for various populations, with little or no consequence to individual students for their test performance. Students' test-taking behaviors in survey assessments, particularly the level of test-taking effort, and their effects on performance have been a long-standing question. This paper presents a procedure to examine test-taking behaviors using response time collected from a National Assessment of Educational Progress (NAEP) computer-based study, referred to as MCBS.Methods: A five-step procedure was proposed to identify rapid-guessing behavior in a more systematic manner. It involves a non-model-based approach that classifies student-item pairs as reflecting either solution behavior or rapid-guessing behavior. Three validity checks were incorporated in the validation step to ensure reasonableness of the time boundaries before further investigation. Results of behavior classification were summarized by three measures to investigate whether and how students' test-taking behaviors related to student characteristics, item characteristics, or both.Results: In the MCBS, the validity checks offered compelling evidence that the recommended threshold-identification method was effective in separating rapid-guessing behavior from solution behavior. A very low percent of rapid-guessing behavior was identified, as compared to existing results for different assessments. For this dataset, rapid-guessing behavior had minimum impact on parameter estimation in the IRT modeling. However, the students clearly exhibited different behaviors when they received items that did not match their performance level. We also found disagreement between students' response-time effort and self reports, but based on the observed data, it is unclear whether the disagreement was related to how the students interpreted the background questions.Conclusions: The paper provides a way to address the issue of identifying rapid-guessing behavior, and sheds light on the question about students' extent of engagement in NAEP and the impact, without relying on students' self evaluation or additional costs in test design. It reveals useful information about test-taking behaviors in a NAEP assessment setting that has not been available in the literature. The procedure is applicable to future standard NAEP assessments, as well as other tests, when timing data are available.