Doctoral Dissertation Research: Preassembly Methods for Cognitive Diagnostic Multistage Adaptive Testing

本博士论文研究项目将推进认知诊断多阶段自适应测试（CD-MST）的测试预组装方法。这种类型的计算机管理的教育测试确定哪些技能的学生已经成功地学习了一个简短的，但可靠的，测试是个性化的每个学生。带有模块预组装功能的CD-MST允许考试开发人员在考试管理之前检查考试表格的质量，并提供有关学生掌握了哪些主题的详细诊断信息以及对他们成绩的评估。然而，只有少数的方法已经提出了在CD-MST的预组装测试。此外，当需要满足其他约束时，使用CD-MST时存在独特的挑战，例如确保测试提供测试所涵盖的所有内容区域的均衡。目前的方法需要严格的假设，不能适应多个测试约束。在这个项目中开发的测试开发战略将推进课堂评估系统，其中测试可以作为学习过程的一部分，而不是作为一种工具，以排名学生的学习成果。为每个学生提供准确，具体和实时的多标准反馈对于有效的学习和补救教学至关重要。采用这种新测试模式的课堂评估系统将在全年提供简短的自适应测试，立即产生符合教育标准的有效和可靠的个性化能力诊断。作为博士论文研究改进奖，提供支持，使有前途的学生建立一个强大的，独立的研究生涯。这个博士论文研究项目将开发一个整体的认知诊断多阶段自适应测试（CD-MST）预组装方法，将同时考虑多种类型的测试约束。为了充分利用CD-MST，关键是要开发不仅包含统计约束（例如，最大化可靠性），而且还有内容约束（例如，确保所有测试内容）而不牺牲估计精度。目前，大多数CD-MST应用都采用了在进行中的测试会话期间组装模块的动态CD-MST。然而，这种方法可能会导致在确保不同版本的等效测试和满足非统计约束的困难，因为测试是在测试会话期间组装。在这个项目中开发的整体方法将适应以前提出的方法，有些不同的情况下：项目选择在一个正在进行的测试会话和预组装的诊断自适应评估。将进行模拟研究，以评估新开发的方法的准确性。以下三个问题将得到解决：1）整体的CD-MST预组装方法是计算可行的？2)整体CD-MST预组装方法在不同的认知诊断模型和测试长度上的表现如何？3)整体CD-MST预组装方法的性能如何随多个内容约束而变化？整体方法的可行性和性能将根据违反的约束的数量、考生分类准确性（即，考生是否被发现具有他们实际拥有的技能），所使用的项目库的比例，以及考生分类的分布。该奖项反映了NSF的法定使命，并被认为值得通过使用基金会的智力价值和更广泛的影响审查标准进行评估来支持。

This doctoral dissertation research project will advance test preassembly methods for Cognitive Diagnostic Multistage Adaptive Testing (CD-MST). This type of computer-administered educational test identifies which skills students have successfully learned with a short, yet reliable, test that is individualized for each student. CD-MST with module preassembly allows test developers to review the quality of test forms before test administration and provides detailed diagnostic information about what topics students have mastered alongside an evaluation of their achievements. However, only a few methods have been proposed to preassemble tests in CD-MST. In addition, there are unique challenges when using CD-MSTs when other constraints need to be met, such as ensuring that the test provides an even balance of all content areas covered by the test. Current methods require strict assumptions and cannot accommodate multiple test constraints. The test development strategy to be developed in this project will advance classroom assessment systems in which tests can serve as a part of learning process rather than as a tool to rank students by learning outcomes. Providing accurate, specific, and real-time feedback on multiple criteria for each student is essential for effective learning and remedial instruction. Classroom assessment systems that adopt this new test mode will provide short adaptive tests throughout the year that instantly yield valid and reliable personalized competency diagnoses aligned with educational standards. As a Doctoral Dissertation Research Improvement award, support is provided to enable a promising student to establish a strong, independent research career.This doctoral dissertation research project will develop a holistic Cognitive Diagnostic Multistage Adaptive Testing (CD-MST) preassembly method that will simultaneously consider numerous types of test constraints. To take full advantage of CD-MST, it is crucial to develop preassembly methods that incorporate not only statistical constraints (e.g., maximizing reliability), but also content constraints (e.g., ensuring all contents tested) without sacrificing estimation precision. Currently, most CD-MST applications have employed on-the-fly CD-MST that assembles modules during an in-progress test session. However, this approach may lead to difficulties in ensuring different versions of equivalent tests and satisfying the non-statistical constraints because the tests are assembled during the testing session. The holistic method to be developed in this project will adapt methods that have been previously proposed for somewhat different contexts: item selection during an in-progress test session and preassembly for diagnostic adaptive assessment. Simulation studies will be conducted to evaluate the accuracy of the newly developed method. The following three questions will be addressed: 1) Is the holistic CD-MST preassembly method computationally feasible? 2) How well does the holistic CD-MST preassembly method perform across different cognitive diagnostic models and test lengths? 3) How does the performance of a holistic CD-MST preassembly method change with multiple content constraints? The feasibility and performance of the holistic method will be evaluated in terms of the number of constraints violated, examinee classification accuracy (i.e., whether examinees are found to have the skill set that they actually have), the proportion of the item pool used, and the distribution of examinees classifications.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.