Doctoral Dissertation Research: Preassembly Methods for Cognitive Diagnostic Multistage Adaptive Testing

博士论文研究：认知诊断多级自适应测试的预组装方法

• 批准号: 2242094
• 负责人: Leah Feuerstahler
• 金额: $ 0.54万
• 依托单位: Fordham University
• 依托单位国家: 美国
• 项目类别: Standard Grant
• 财政年份: 2023
• 资助国家: 美国
• 起止时间: 2023-03-01 至 2024-02-29
• 项目状态: 已结题
• 来源: https://www.nsf.gov/awardsearch/showAward?AWD_ID=2242094&HistoricalAwards=false
• 关键词: Doctoral; Dissertation; Research; Preassembly; Methods

This doctoral dissertation research project will advance test preassembly methods for Cognitive Diagnostic Multistage Adaptive Testing (CD-MST). This type of computer-administered educational test identifies which skills students have successfully learned with a short, yet reliable, test that is individualized for each student. CD-MST with module preassembly allows test developers to review the quality of test forms before test administration and provides detailed diagnostic information about what topics students have mastered alongside an evaluation of their achievements. However, only a few methods have been proposed to preassemble tests in CD-MST. In addition, there are unique challenges when using CD-MSTs when other constraints need to be met, such as ensuring that the test provides an even balance of all content areas covered by the test. Current methods require strict assumptions and cannot accommodate multiple test constraints. The test development strategy to be developed in this project will advance classroom assessment systems in which tests can serve as a part of learning process rather than as a tool to rank students by learning outcomes. Providing accurate, specific, and real-time feedback on multiple criteria for each student is essential for effective learning and remedial instruction. Classroom assessment systems that adopt this new test mode will provide short adaptive tests throughout the year that instantly yield valid and reliable personalized competency diagnoses aligned with educational standards. As a Doctoral Dissertation Research Improvement award, support is provided to enable a promising student to establish a strong, independent research career.This doctoral dissertation research project will develop a holistic Cognitive Diagnostic Multistage Adaptive Testing (CD-MST) preassembly method that will simultaneously consider numerous types of test constraints. To take full advantage of CD-MST, it is crucial to develop preassembly methods that incorporate not only statistical constraints (e.g., maximizing reliability), but also content constraints (e.g., ensuring all contents tested) without sacrificing estimation precision. Currently, most CD-MST applications have employed on-the-fly CD-MST that assembles modules during an in-progress test session. However, this approach may lead to difficulties in ensuring different versions of equivalent tests and satisfying the non-statistical constraints because the tests are assembled during the testing session. The holistic method to be developed in this project will adapt methods that have been previously proposed for somewhat different contexts: item selection during an in-progress test session and preassembly for diagnostic adaptive assessment. Simulation studies will be conducted to evaluate the accuracy of the newly developed method. The following three questions will be addressed: 1) Is the holistic CD-MST preassembly method computationally feasible? 2) How well does the holistic CD-MST preassembly method perform across different cognitive diagnostic models and test lengths? 3) How does the performance of a holistic CD-MST preassembly method change with multiple content constraints? The feasibility and performance of the holistic method will be evaluated in terms of the number of constraints violated, examinee classification accuracy (i.e., whether examinees are found to have the skill set that they actually have), the proportion of the item pool used, and the distribution of examinees classifications.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.

本博士论文研究项目将推进认知诊断多阶段自适应测验(CD-MST)的测验预组装方法。这种类型的计算机管理的教育测试通过针对每个学生的简短而可靠的测试来确定学生成功地学习了哪些技能。带有模块预组装的CD-MST允许测试开发人员在考试管理之前审查测试表的质量，并提供关于学生掌握了哪些主题的详细诊断信息，以及对他们的成绩的评估。然而，在CD-MST中只提出了几种预装配测试的方法。此外，当需要满足其他限制时，使用CD-MST还有独特的挑战，例如确保测试提供测试覆盖的所有内容区域的均衡平衡。目前的方法需要严格的假设，不能适应多个测试约束。本项目将开发的测试开发策略将促进课堂评估系统的发展，在该系统中，测试可以作为学习过程的一部分，而不是作为根据学习结果对学生进行排名的工具。为每个学生提供关于多种标准的准确、具体和实时的反馈，对于有效的学习和辅导是必不可少的。采用这种新测试模式的课堂评估系统将在全年提供短期的适应性测试，立即产生与教育标准一致的有效和可靠的个性化能力诊断。作为博士论文研究改进奖，提供支持，使有前途的学生建立一个强大的，独立的研究生涯。本博士论文研究项目将开发一种整体认知诊断多阶段自适应测试(CD-MST)预组装方法，将同时考虑多种类型的测试约束。为了充分利用CD-MST，至关重要的是开发不仅包括统计约束(例如，最大限度地提高可靠性)，而且包括内容约束(例如，确保所有测试的内容)而不牺牲估计精度的预汇编方法。目前，大多数CD-MST应用程序都采用了在进行中的测试过程中组装模块的动态CD-MST。然而，这种方法可能会导致在确保不同版本的等价测试和满足非统计约束方面存在困难，因为测试是在测试过程中组装的。本项目将开发的整体方法将使以前提出的方法适用于略有不同的情况：在进行中的测试期间进行项目选择，以及为诊断适应性评估预先组装。将进行模拟研究，以评估新开发的方法的准确性。以下三个问题将被讨论：1)整体CD-MST预编译方法在计算上可行吗？2)整体CD-MST预编译方法在不同认知诊断模型和测试长度上的表现如何？3)整体CD-MST预编译方法的性能在多个内容约束下如何变化？整体方法的可行性和表现将根据违反的限制条件的数量、考生分类的准确性(即考生是否被发现具有他们实际拥有的技能集)、使用的题库的比例以及考生分类的分布来评估。该奖项反映了NSF的法定使命，并通过使用基金会的智力优势和更广泛的影响审查标准进行评估，被认为值得支持。