Collaborative Research: Developing and Evaluating Assessments of Problem-Solving in Computer Adaptive Testing Environments

四十多年来，解决问题一直是K-12数学教育的优先事项，并反映在41个州以某种形式采用的共同核心国家数学标准（CCSSM）倡议中。从广义上讲，问题解决涉及学生在解决具有智力挑战性的数学任务时所从事的数学实践。在先前的研究中，开发并验证了与3-5年级CCSSM相一致的解决问题措施，以补充6-8年级先前建立的解决问题措施。解决问题的措施评估学生的解决问题的表现在CCSSM数学内容和实践的背景下。本研究扩大了问题解决措施的使用范围和分数解释。该项目的工作推进数学问题解决评估到计算机自适应测试。与静态测试相比，计算机自适应测试可以更精确和有效地针对学生的能力。很少有评估学生数学问题解决能力的措施使用这种技术。较短的测试需要更少的课堂评估时间比目前的纸笔解决问题的措施，并增加课堂教学时间。计算机自适应问题解决方法具有较强的信度和效度，可以有效抑制考生的疲劳。最后，该项目将使用客观的标准制定方法对目前6-8年级的文书进行基准测试，从而可以通过与内容相关的反馈来改进分数解释。学生和班级报告的即时结果将通过计算机自适应测试系统产生，使教师能够修改教学以改善学生的学习，这个五年期项目旨在促进计算机自适应测试和评估发展在数学教学中的使用。该项目采用了迭代和知情的设计科学为基础的方法，以及在项目开发和验证过程中使用Rasch建模进行心理测量分析。该项目旨在：（a）对先前制定的6-8年级解决问题的措施进行基准测试;（B）为每项措施制定、校准和验证标准参考的计算机自适应测试;（c）构建学生和班级级别的分数报告，以纳入计算机自适应测试系统;（d）调查教师在STEM学习环境中实施、解释和使用评估和结果的能力。该项目解决了以下一组研究问题：（RQ 1）什么基准性能标准定义不同的熟练程度水平的解决问题的措施，为每个年级？(RQ2)为电脑自适应测验试题库所开发的新解题测量项目，其心理测量特性为何？(RQ3)在新的问题解决测量项目上，学生群体之间是否存在显著的项目漂移？(RQ4)在计算机自适应测试系统中，问题解决测量项目校准的稳定程度如何？(RQ5)教师和学生对新的解题措施项目、电脑自适应测试平台和报告系统有何改进建议？(RQ6)教师在多大程度上与评估信息互动、感知和理解这些信息？以及（RQ 7）在线学习模块是否能够培养教师在STEM学习环境中解决问题的措施、计算机自适应测试的实施、解释和使用学生评估结果的能力？本研究采用实验设计，探讨教师在电脑自适应测验系统中执行、诠释及运用问题解决措施的能力。该项目有可能影响该领域，为学区和研究人员提供一种手段，以有效和高效地评估学生在一段时间内解决数学问题的表现或随着时间的推移而增长;解决未来的在线学习需求;通过更准确地了解学生的优势，减少用于评估的课堂时间，改善课堂教学。旨在通过研究和开发创新资源，模型和工具，显着提高学前班学生和教师的科学，技术，工程和数学（STEM）的学习和教学。DRK-12项目中的项目建立在STEM教育的基础研究以及为拟议项目提供理论和经验依据的先前研究和开发工作的基础上。该奖项反映了NSF的法定使命，并通过使用基金会的知识价值和更广泛的影响审查标准进行评估，被认为值得支持。

Problem solving has been a priority within K-12 mathematics education for over four decades and is reflected throughout the Common Core State Standards for Mathematics (CCSSM) initiative, which have been adopted in some form by 41 states. Broadly defined, problem solving involves the mathematical practices in which students engage as they solve intellectually-challenging mathematical tasks. In prior research, problem-solving measures aligned to CCSSM for grades 3-5 were developed and validated to supplement previously established problem-solving measures in grades 6-8. The problem-solving measures assess students’ problem-solving performance within the context of CCSSM math content and practices. This project expands the scope of the problem-solving measures use and score interpretation. The project work advances mathematical problem-solving assessments into computer adaptive testing. Computer adaptive testing allows for more precise and efficient targeting of student ability compared to static tests. Few measures designed to assess students’ mathematical problem-solving ability use this technology. Shorter tests require less in-class time for assessment than current paper-pencil problem-solving measures and increase classroom instruction time. The computer-adaptive problem-solving measures have sufficient reliability and strong validity evidence, and may limit test-taker fatigue. Finally, the project will benchmark current grades 6-8 instruments using an objective standard-setting method, which allows for improved score interpretations with content-related feedback. Immediate results of student- and class-level reports will be produced through the computer adaptive testing system allowing for teachers to modify instruction to improve students’ learning.This five-year project aims to advance the use of computer adaptive testing and assessment development for use in mathematics instruction. The project applies an iterative and stakeholder-informed design science-based methodology as well as employs the use of Rasch modeling for the psychometric analysis during item development and validation. The project aims to: (a) benchmark the previously established grades 6-8 problem-solving measures; (b) develop, calibrate, and validate criterion-referenced computer adaptive testing for each measure; (c) construct student- and class-level score reports for integration into the computer adaptive testing system; and (d) investigate teachers’ capacity for implementing, interpreting, and using the assessments and results in STEM learning settings. The project addresses the following set of research questions: (RQ1) What benchmark performance standards define different proficiency levels on problem-solving measures for each grade level? (RQ2) What are the psychometric properties of new problem-solving measures items developed for the computer adaptive testing item bank? (RQ3) Is there significant item drift across student populations on the new problem-solving measure items? (RQ4) To what extent are problem-solving measures item calibrations stable within the computer adaptive testing system? (RQ5) What recommendations for improvements do teachers and students have for the new problem-solving measures items, computer adaptive testing platform and reporting system, if any? (RQ6) To what extent do teachers interact with, perceive, and make sense of the assessment information generated for use in practice? and (RQ7) Does an online learning module build teacher capacity for problem solving measures, computer adaptive testing implementation, interpretation, and use of student assessment outcomes in STEM learning settings? An experimental design will be utilized to investigate teachers’ capacity for implementing, interpreting, and using problem solving measures in a computer adaptive testing system. The project has the potential to impact the field by providing school districts and researchers a means to assess students’ mathematical problem-solving performance at one time or growth over time efficiently and effectively; address future online learning needs; and improve classroom teaching through more precise information about students’ strengths with less class time focused on assessment.The Discovery Research preK-12 program (DRK-12) seeks to significantly enhance the learning and teaching of science, technology, engineering and mathematics (STEM) by preK-12 students and teachers, through research and development of innovative resources, models and tools. Projects in the DRK-12 program build on fundamental research in STEM education and prior research and development efforts that provide theoretical and empirical justification for proposed projects.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.