The Item Bank Calibration and Replenishment for Computerized Adaptive Testing in Small Scale Assessments: Method, Theory, and Application

该研究项目将推进小规模评估中计算机化自适应测试（CAT）题库校准和补充的统计估计方法。 CAT已经成为一种强大的评估工具，并已被应用于教育测试，生活质量测量，健康相关测量和工业环境测试领域。与传统的纸笔测试不同，CAT允许个性化评估。然而，CAT的应用仍然局限于小规模的评估场景，如在教室或商业日常。本计画将开发一系列统计评估方法、理论、演算法与软体，以加速电脑辅助教学的发展，并促进其在个人化教育评估与学习上的应用。该项目将解决教育研究中的一个基本挑战：缺乏有效的统计方法来科普有限的样本量和缺失的数据，特别是在低风险评估环境中。预计这些成果将推动教育和心理学领域的发展，并对社会产生积极影响。研究人员将把新方法应用于教育数据集，并开发可直接适用于教育工作者和应用研究人员的公开软件。研究生和本科生将参与这项研究的进行。本研究将开发新的统计方法和理论，以解决项目反应理论（IRT）模型校准的挑战，小样本，稀疏的响应数据，以及缺乏计算效率高的在线估计程序与理论保证。该项目将开发一种基于同质结构学习的IRT模型族的降维方法，它属于一类统计和机器学习方法，旨在通过发现低维结构来总结有用的信息。 一个在线推理的方法，借用了一些最近的发展基于随机梯度下降的在线估计和推理的智慧也将被开发。 调查人员将通过一组模拟研究，模拟小规模评估的不同方面，并分析两个真实的数据集，从基于计算机的课堂评估和大规模的CAT验证这些开发的方法。该奖项反映了NSF的法定使命，并已被认为是值得通过使用基金会的智力价值和更广泛的影响审查标准进行评估的支持。

This research project will advance statistical estimation methods for computerized adaptive testing (CAT) item bank calibration and replenishment in small-scale assessments. CAT has emerged as a powerful assessment tool and has been applied to the field of educational testing, quality of life measurement, health related measurement, and testing in industrial settings. Different from the traditional paper-pencil test, CAT allows for personalized assessment. However, the application of CAT remains limited in small-scale assessment scenarios, such as in classrooms or business daily routines. This project will develop a series of statistical estimation methods, theories, algorithms, and software aimed at accelerating the development of CAT and promoting its applications in personalized educational assessment and learning. The project will address a fundamental challenge in educational research: the lack of efficient statistical methods to cope with limited samples sizes and missing data, especially in low-stakes assessment settings. The outcomes are expected to advance the fields of education and psychology and to positively impact society. The investigators will apply the new methods to educational data sets and develop publicly available software that will be directly applicable for educators and applied researchers. Graduate and undergraduate students will participate in the conduct of this research. This research project will develop new statistical methods and theories to address the challenges of Item Response Theory (IRT) model calibration with small samples, sparse response data, and the lack of computationally efficient online estimation procedures with theoretical guarantee. The project will develop a dimension reduction method for a family of IRT models based on homogeneity structure learning, which belongs to a class of statistical and machine learning methods that aim to summarize useful information by discovering low-dimensional structures. An online inference approach that borrows the wisdom of some recent developments in stochastic gradient descent based online estimation and inference also will be developed. The investigators will validate these developed approaches through a set of simulation studies that mimic different aspects of small-scale assessments and analyze two real data sets from a computer-based classroom assessment and a large-scale CAT.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.