Improving Computerized Adaptive Testing In the United States

改善美国的计算机化自适应测试

• 批准号: 0613025
• 负责人: Hua-Hua Chang
• 金额: $ 2.19万
• 依托单位: University of Illinois at Urbana-Champaign
• 依托单位国家: 美国
• 项目类别: Standard Grant
• 财政年份: 2005
• 资助国家: 美国
• 起止时间: 2005-08-16 至 2007-07-31
• 项目状态: 已结题
• 来源: https://www.nsf.gov/awardsearch/showAward?AWD_ID=0613025&HistoricalAwards=false
• 关键词: Improving; Computerized; Adaptive; Testing; United

While the implementation of computerized adaptive testing (CAT) has many advantages, many issues related to CATs are not well understood. This project will study three specific issues in development and implementation of CAT: (1) compatibility between CAT and "paper and pencil" (P&P) tests, (2) test security and item pool usage, and (3) how to calibrate test items in large quantities efficiently and economically. With respect to compatibility between CAT and "paper and pencil" (P&P) tests, it has been widely reported that some students get much lower scores than they would if an alternative P&P version were given. However, examinees currently required to take Graduate Record Examination (GRE) in the United States, for instance, are not given a choice between the standard P&P version of the tests and the CAT versions. Without effective remedial measures, the credibility of CAT could be significantly undermined. This project proposes to modify the statistical procedure used for CAT item selection by incorporating some advanced analytic techniques. It is expected that the analytic and simulation results will show that weighting likelihood score may alleviate the problem of underestimation. With respect to test security and item pool usage, in current operational CATs, computers tend to select certain types of items too frequently, making item exposure rates quite uneven. This project will show that test security and the underestimation problem discussed in the research on CAT and P&P tests are closely related. It is also expected that the project will show that the alpha-stratified approach proposed by Chang and Ying in 1999 tends to improve both the underestimation and test security. With respect to calibrating test items in large quantities efficiently and economically, administration of CATs requires very large item pools. Fortunately, CAT provides great potential to large-scale calibration during on-line testing. This project will explore the development of on-line calibration in CAT.CAT has become a popular mode of educational assessment in the United States. Examples of large scale CATs include the Graduate Record Examination (GRE), the Graduate Management Admission Test (GMAT), the National Council of State Boards of Nursing, and the Armed Services Vocational Aptitude Battery (ASVAB). Findings from this research project may speed up the process of improvement over current item selection algorithms. Because many CATs are high-stakes examinations, improving their test reliabilities will greatly benefit society.

虽然计算机自适应测验(CAT)的实施有很多优点，但与CAT相关的许多问题还没有得到很好的理解。本项目将研究CAT开发和实施中的三个具体问题：(1)CAT与“纸笔”(P&amp；P；P)测试之间的兼容性，(2)测试安全性和题库使用，以及(3)如何高效和经济地大量校准测试项目。关于CAT和“纸笔”(P&amp；P；P)测试之间的兼容性，据广泛报道，一些学生得到的分数比如果给他们一个替代P&P版本的话要低得多。然而，例如，目前美国要求参加研究生入学考试(GRE)的考生，不能在标准的P&P版本和CAT版本之间进行选择。如果没有有效的补救措施，《禁止酷刑公约》的可信度可能会受到严重损害。该项目建议通过采用一些先进的分析技术来修改用于CAT项目选择的统计程序。预计分析和仿真结果将表明，加权似然分数可以缓解低估估计的问题。在测试安全性和题库使用方面，在当前的操作CAT中，计算机倾向于过于频繁地选择某些类型的题项，使得项目曝光率相当不均匀。这个项目将表明，测试安全性与CAT和P&P测试研究中讨论的低估问题密切相关。预计该项目将表明，Chang和Ying在1999年提出的阿尔法分层方法倾向于改善低估和测试安全性。就大量、高效和经济地校准测试项目而言，CATS的管理需要非常大的题库。幸运的是，CAT为在线测试中的大规模校准提供了巨大的潜力。该项目将探索CAT在线校准的发展。CAT已成为美国流行的教育评估模式。大规模CAST的例子包括研究生入学考试(GRE)、研究生管理入学考试(GMAT)、国家护理委员会和军队职业能力评估小组(ASVAB)。这一研究项目的发现可能会加快改进现有项目选择算法的进程。由于很多猫都是高风险的考试，提高它们的测试可靠性将极大地造福社会。