The Item Bank Calibration and Replenishment for Computerized Adaptive Testing in Small Scale Assessments: Method, Theory, and Application

本研究项目将推进计算机自适应测试（CAT）题库在小规模评估中的校准和补充的统计估计方法。CAT已成为一种强大的评估工具，并已应用于教育测试、生活质量测量、健康相关测量和工业环境测试领域。与传统的纸笔测试不同，CAT允许个性化评估。然而，CAT在小规模评估场景中的应用仍然有限，例如在教室或日常业务中。本项目将开发一系列统计估计方法、理论、算法和软件，旨在加速CAT的发展，促进其在个性化教育评估和学习中的应用。该项目将解决教育研究中的一个基本挑战：缺乏有效的统计方法来处理有限的样本量和缺失的数据，特别是在低风险评估环境中。研究结果有望推动教育和心理学领域的发展，并对社会产生积极影响。研究人员将把新方法应用于教育数据集，并开发可直接适用于教育工作者和应用研究人员的公开软件。研究生和本科生将参与本研究的进行。本研究项目将开发新的统计方法和理论，以解决项目反应理论（IRT）模型在小样本、稀疏响应数据下的校准以及缺乏具有理论保证的计算效率的在线估计程序的挑战。该项目将为一组基于同质结构学习的IRT模型开发一种降维方法，该方法属于一类统计和机器学习方法，旨在通过发现低维结构来总结有用的信息。本文还将开发一种基于随机梯度下降的在线估计和推理的在线推理方法。研究人员将通过一系列模拟研究来验证这些开发的方法，这些研究模拟小规模评估的不同方面，并分析来自基于计算机的课堂评估和大规模CAT的两个真实数据集。该奖项反映了美国国家科学基金会的法定使命，并通过使用基金会的知识价值和更广泛的影响审查标准进行评估，被认为值得支持。

This research project will advance statistical estimation methods for computerized adaptive testing (CAT) item bank calibration and replenishment in small-scale assessments. CAT has emerged as a powerful assessment tool and has been applied to the field of educational testing, quality of life measurement, health related measurement, and testing in industrial settings. Different from the traditional paper-pencil test, CAT allows for personalized assessment. However, the application of CAT remains limited in small-scale assessment scenarios, such as in classrooms or business daily routines. This project will develop a series of statistical estimation methods, theories, algorithms, and software aimed at accelerating the development of CAT and promoting its applications in personalized educational assessment and learning. The project will address a fundamental challenge in educational research: the lack of efficient statistical methods to cope with limited samples sizes and missing data, especially in low-stakes assessment settings. The outcomes are expected to advance the fields of education and psychology and to positively impact society. The investigators will apply the new methods to educational data sets and develop publicly available software that will be directly applicable for educators and applied researchers. Graduate and undergraduate students will participate in the conduct of this research. This research project will develop new statistical methods and theories to address the challenges of Item Response Theory (IRT) model calibration with small samples, sparse response data, and the lack of computationally efficient online estimation procedures with theoretical guarantee. The project will develop a dimension reduction method for a family of IRT models based on homogeneity structure learning, which belongs to a class of statistical and machine learning methods that aim to summarize useful information by discovering low-dimensional structures. An online inference approach that borrows the wisdom of some recent developments in stochastic gradient descent based online estimation and inference also will be developed. The investigators will validate these developed approaches through a set of simulation studies that mimic different aspects of small-scale assessments and analyze two real data sets from a computer-based classroom assessment and a large-scale CAT.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.