Collaborative Research: Developing and Evaluating Assessments of Problem-Solving in Computer Adaptive Testing Environments

四十多年来，解决问题一直是K-12数学教育的重点，并在41个州以某种形式采用的“共同核心州数学标准”（CCSSM）倡议中得到了体现。从广义上讲，解决问题涉及学生在解决具有智力挑战性的数学任务时所进行的数学实践。在之前的研究中，针对3-5年级的CCSSM制定并验证了与CCSSM相一致的问题解决措施，以补充先前建立的6-8年级问题解决措施。问题解决措施评估学生在CCSSM数学内容和实践背景下的问题解决表现。这个项目扩展了问题解决措施的使用和分数解释的范围。该项目工作将数学问题解决评估推进到计算机自适应测试中。与静态测试相比，计算机自适应测试可以更精确、更有效地瞄准学生的能力。很少有评估学生数学解决问题能力的方法使用这种技术。与目前的纸笔解决问题的方法相比，更短的测试需要更少的课堂时间进行评估，并增加了课堂教学时间。计算机自适应解决问题的措施有足够的信度和强有力的效度证据，并可能限制考生疲劳。最后，该项目将使用客观标准制定方法对当前6-8级的工具进行基准测试，该方法允许通过与内容相关的反馈改进分数解释。学生和班级报告的即时结果将通过计算机自适应测试系统产生，允许教师修改教学以改善学生的学习。这个为期五年的项目旨在促进计算机自适应测试和评估开发在数学教学中的应用。该项目采用迭代和利益相关者知情的科学设计方法，并在项目开发和验证过程中使用Rasch模型进行心理测量分析。该项目的目的是：(a)以先前制订的6至8年级解决问题的措施为基准；(b)为每项测量开发、校准和验证参考标准的计算机自适应测试；(c)编制学生及班级的成绩报告，以便纳入电脑自适应测试系统；(d)调查教师在STEM学习环境中实施、解释和使用评估和结果的能力。该项目涉及以下一系列研究问题：（RQ1）哪些基准表现标准定义了每个年级解决问题措施的不同熟练程度？（RQ2）为计算机自适应测试题库开发的新问题解决测量项目的心理测量特性是什么？（RQ3）在新的问题解决测量项目上，学生群体之间是否存在显著的项目漂移？在计算机自适应测试系统中，解决问题的测量项目校准在多大程度上是稳定的？（RQ5）教师和学生对新的问题解决措施项目、计算机自适应测试平台和报告系统有什么改进建议？（RQ6）教师在多大程度上与为实践使用而产生的评估信息进行互动、感知和理解？（RQ7）在线学习模块是否能够培养教师在STEM学习环境中解决问题、实施计算机自适应测试、解释和使用学生评估结果方面的能力？一个实验设计将被用来调查教师在计算机自适应测试系统中执行、解释和使用问题解决措施的能力。该项目有可能影响该领域，为学区和研究人员提供了一种评估学生一次性解决数学问题的能力或随着时间的推移而高效增长的方法；解决未来在线学习需求；通过更准确地了解学生的优势，减少课堂时间的评估，改善课堂教学。探索研究preK-12项目（DRK-12）旨在通过研究和开发创新资源、模型和工具，显著提高preK-12学生和教师对科学、技术、工程和数学（STEM）的学习和教学。DRK-12计划中的项目建立在STEM教育的基础研究和先前的研究和开发工作的基础上，为拟议的项目提供了理论和实证依据。该奖项反映了美国国家科学基金会的法定使命，并通过使用基金会的知识价值和更广泛的影响审查标准进行评估，被认为值得支持。

Problem solving has been a priority within K-12 mathematics education for over four decades and is reflected throughout the Common Core State Standards for Mathematics (CCSSM) initiative, which have been adopted in some form by 41 states. Broadly defined, problem solving involves the mathematical practices in which students engage as they solve intellectually-challenging mathematical tasks. In prior research, problem-solving measures aligned to CCSSM for grades 3-5 were developed and validated to supplement previously established problem-solving measures in grades 6-8. The problem-solving measures assess students’ problem-solving performance within the context of CCSSM math content and practices. This project expands the scope of the problem-solving measures use and score interpretation. The project work advances mathematical problem-solving assessments into computer adaptive testing. Computer adaptive testing allows for more precise and efficient targeting of student ability compared to static tests. Few measures designed to assess students’ mathematical problem-solving ability use this technology. Shorter tests require less in-class time for assessment than current paper-pencil problem-solving measures and increase classroom instruction time. The computer-adaptive problem-solving measures have sufficient reliability and strong validity evidence, and may limit test-taker fatigue. Finally, the project will benchmark current grades 6-8 instruments using an objective standard-setting method, which allows for improved score interpretations with content-related feedback. Immediate results of student- and class-level reports will be produced through the computer adaptive testing system allowing for teachers to modify instruction to improve students’ learning.This five-year project aims to advance the use of computer adaptive testing and assessment development for use in mathematics instruction. The project applies an iterative and stakeholder-informed design science-based methodology as well as employs the use of Rasch modeling for the psychometric analysis during item development and validation. The project aims to: (a) benchmark the previously established grades 6-8 problem-solving measures; (b) develop, calibrate, and validate criterion-referenced computer adaptive testing for each measure; (c) construct student- and class-level score reports for integration into the computer adaptive testing system; and (d) investigate teachers’ capacity for implementing, interpreting, and using the assessments and results in STEM learning settings. The project addresses the following set of research questions: (RQ1) What benchmark performance standards define different proficiency levels on problem-solving measures for each grade level? (RQ2) What are the psychometric properties of new problem-solving measures items developed for the computer adaptive testing item bank? (RQ3) Is there significant item drift across student populations on the new problem-solving measure items? (RQ4) To what extent are problem-solving measures item calibrations stable within the computer adaptive testing system? (RQ5) What recommendations for improvements do teachers and students have for the new problem-solving measures items, computer adaptive testing platform and reporting system, if any? (RQ6) To what extent do teachers interact with, perceive, and make sense of the assessment information generated for use in practice? and (RQ7) Does an online learning module build teacher capacity for problem solving measures, computer adaptive testing implementation, interpretation, and use of student assessment outcomes in STEM learning settings? An experimental design will be utilized to investigate teachers’ capacity for implementing, interpreting, and using problem solving measures in a computer adaptive testing system. The project has the potential to impact the field by providing school districts and researchers a means to assess students’ mathematical problem-solving performance at one time or growth over time efficiently and effectively; address future online learning needs; and improve classroom teaching through more precise information about students’ strengths with less class time focused on assessment.The Discovery Research preK-12 program (DRK-12) seeks to significantly enhance the learning and teaching of science, technology, engineering and mathematics (STEM) by preK-12 students and teachers, through research and development of innovative resources, models and tools. Projects in the DRK-12 program build on fundamental research in STEM education and prior research and development efforts that provide theoretical and empirical justification for proposed projects.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.