Improving Computerized Adaptive Testing In the United States

虽然计算机化自适应测试（CAT）的实施有许多优点，但与CAT相关的许多问题还没有得到很好的理解。 本研究将探讨CAT发展与实施中的三个具体问题：（1）CAT与纸笔测验的相容性;（2）测验的安全性与试题库的使用;（3）如何有效且经济地大量校正试题。 关于CAT和“纸笔”（P P）测试之间的兼容性，据广泛报道，一些学生的分数比他们在另一种P P版本的情况下得到的分数要低得多。 然而，例如，美国目前要求参加研究生入学考试（GRE）的考生不能在标准P P版本的考试和CAT版本的考试之间做出选择。 如果不采取有效的补救措施，《禁止酷刑公约》的可信度可能会受到严重损害。 本计画提出将先进的分析技术应用于电脑辅助测验项目选择的统计程序。 预计分析和模拟结果将表明，加权似然得分可以缓解低估的问题。 关于测试安全性和项目池的使用，在当前的操作CAT中，计算机倾向于过于频繁地选择某些类型的项目，使得项目暴露率非常不均匀。 本项目将表明，CAT和P P测试研究中讨论的低估问题与测试安全性密切相关。 预计该项目将表明，Chang和Ying在1999年提出的α分层方法倾向于改善低估和测试安全性。 关于有效和经济地校准大量测试项目，CAT的管理需要非常大的项目池。 幸运的是，CAT为在线测试期间的大规模校准提供了巨大的潜力。 该项目将探索CAT在线校准的发展。CAT已成为美国流行的教育评估模式。 大规模CAT的例子包括研究生入学考试（GRE），研究生管理入学考试（GMAT），国家护理委员会的国家理事会和武装部队职业能力测试（ASVAB）。 从这个研究项目的结果可能会加快目前的项目选择算法的改进过程。 由于许多CAT是高风险的考试，提高其考试的可靠性将大大有利于社会。

While the implementation of computerized adaptive testing (CAT) has many advantages, many issues related to CATs are not well understood. This project will study three specific issues in development and implementation of CAT: (1) compatibility between CAT and "paper and pencil" (P&P) tests, (2) test security and item pool usage, and (3) how to calibrate test items in large quantities efficiently and economically. With respect to compatibility between CAT and "paper and pencil" (P&P) tests, it has been widely reported that some students get much lower scores than they would if an alternative P&P version were given. However, examinees currently required to take Graduate Record Examination (GRE) in the United States, for instance, are not given a choice between the standard P&P version of the tests and the CAT versions. Without effective remedial measures, the credibility of CAT could be significantly undermined. This project proposes to modify the statistical procedure used for CAT item selection by incorporating some advanced analytic techniques. It is expected that the analytic and simulation results will show that weighting likelihood score may alleviate the problem of underestimation. With respect to test security and item pool usage, in current operational CATs, computers tend to select certain types of items too frequently, making item exposure rates quite uneven. This project will show that test security and the underestimation problem discussed in the research on CAT and P&P tests are closely related. It is also expected that the project will show that the alpha-stratified approach proposed by Chang and Ying in 1999 tends to improve both the underestimation and test security. With respect to calibrating test items in large quantities efficiently and economically, administration of CATs requires very large item pools. Fortunately, CAT provides great potential to large-scale calibration during on-line testing. This project will explore the development of on-line calibration in CAT.CAT has become a popular mode of educational assessment in the United States. Examples of large scale CATs include the Graduate Record Examination (GRE), the Graduate Management Admission Test (GMAT), the National Council of State Boards of Nursing, and the Armed Services Vocational Aptitude Battery (ASVAB). Findings from this research project may speed up the process of improvement over current item selection algorithms. Because many CATs are high-stakes examinations, improving their test reliabilities will greatly benefit society.