Collaborative Research: Developing and Evaluating Assessments of Problem-Solving in Computer Adaptive Testing Environments

协作研究：在计算机自​​适应测试环境中开发和评估解决问题的评估

• 批准号: 2101026
• 负责人: Toni May-Sondergeld
• 金额: $ 219.57万
• 依托单位: Drexel University
• 依托单位国家: 美国
• 项目类别: Continuing Grant
• 财政年份: 2021
• 资助国家: 美国
• 起止时间: 2021-08-01 至 2026-07-31
• 项目状态: 未结题
• 来源: https://www.nsf.gov/awardsearch/showAward?AWD_ID=2101026&HistoricalAwards=false
• 关键词: Collaborative; Research; Developing; Evaluating; Assessments

Problem solving has been a priority within K-12 mathematics education for over four decades and is reflected throughout the Common Core State Standards for Mathematics (CCSSM) initiative, which have been adopted in some form by 41 states. Broadly defined, problem solving involves the mathematical practices in which students engage as they solve intellectually-challenging mathematical tasks. In prior research, problem-solving measures aligned to CCSSM for grades 3-5 were developed and validated to supplement previously established problem-solving measures in grades 6-8. The problem-solving measures assess students’ problem-solving performance within the context of CCSSM math content and practices. This project expands the scope of the problem-solving measures use and score interpretation. The project work advances mathematical problem-solving assessments into computer adaptive testing. Computer adaptive testing allows for more precise and efficient targeting of student ability compared to static tests. Few measures designed to assess students’ mathematical problem-solving ability use this technology. Shorter tests require less in-class time for assessment than current paper-pencil problem-solving measures and increase classroom instruction time. The computer-adaptive problem-solving measures have sufficient reliability and strong validity evidence, and may limit test-taker fatigue. Finally, the project will benchmark current grades 6-8 instruments using an objective standard-setting method, which allows for improved score interpretations with content-related feedback. Immediate results of student- and class-level reports will be produced through the computer adaptive testing system allowing for teachers to modify instruction to improve students’ learning.This five-year project aims to advance the use of computer adaptive testing and assessment development for use in mathematics instruction. The project applies an iterative and stakeholder-informed design science-based methodology as well as employs the use of Rasch modeling for the psychometric analysis during item development and validation. The project aims to: (a) benchmark the previously established grades 6-8 problem-solving measures; (b) develop, calibrate, and validate criterion-referenced computer adaptive testing for each measure; (c) construct student- and class-level score reports for integration into the computer adaptive testing system; and (d) investigate teachers’ capacity for implementing, interpreting, and using the assessments and results in STEM learning settings. The project addresses the following set of research questions: (RQ1) What benchmark performance standards define different proficiency levels on problem-solving measures for each grade level? (RQ2) What are the psychometric properties of new problem-solving measures items developed for the computer adaptive testing item bank? (RQ3) Is there significant item drift across student populations on the new problem-solving measure items? (RQ4) To what extent are problem-solving measures item calibrations stable within the computer adaptive testing system? (RQ5) What recommendations for improvements do teachers and students have for the new problem-solving measures items, computer adaptive testing platform and reporting system, if any? (RQ6) To what extent do teachers interact with, perceive, and make sense of the assessment information generated for use in practice? and (RQ7) Does an online learning module build teacher capacity for problem solving measures, computer adaptive testing implementation, interpretation, and use of student assessment outcomes in STEM learning settings? An experimental design will be utilized to investigate teachers’ capacity for implementing, interpreting, and using problem solving measures in a computer adaptive testing system. The project has the potential to impact the field by providing school districts and researchers a means to assess students’ mathematical problem-solving performance at one time or growth over time efficiently and effectively; address future online learning needs; and improve classroom teaching through more precise information about students’ strengths with less class time focused on assessment.The Discovery Research preK-12 program (DRK-12) seeks to significantly enhance the learning and teaching of science, technology, engineering and mathematics (STEM) by preK-12 students and teachers, through research and development of innovative resources, models and tools. Projects in the DRK-12 program build on fundamental research in STEM education and prior research and development efforts that provide theoretical and empirical justification for proposed projects.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.

40多年来，解决问题一直是K-12数学教育的优先事项，反映在已被41个州以某种形式通过的共同核心国家数学标准(CCSSM)倡议中。广义的问题解决包括学生在解决具有智力挑战的数学任务时所进行的数学实践。在以前的研究中，3-5年级的问题解决措施与CCSSM一致，并得到了验证，以补充先前建立的6-8年级的问题解决措施。问题解决措施在CCSSM数学内容和练习的背景下评估学生的问题解决表现。该项目扩大了问题解决措施的使用范围和分数解释。该项目工作将数学问题解决评估推进到计算机适应性测试中。与静态测试相比，计算机自适应测试可以更精确、更有效地定位学生的能力。很少有评估学生数学问题解决能力的措施使用这项技术。与目前纸笔解决问题的方法相比，更短的测试需要更少的课堂时间来进行评估，并增加课堂教学时间。计算机自适应问题解决措施具有足够的信度和较强的效度证据，可以限制考生的疲劳。最后，该项目将使用客观的标准制定方法对当前6-8年级的工具进行基准测试，这种方法允许通过与内容相关的反馈改进分数解释。这项为期五年的计划，将透过电脑适配测验系统即时产生学生和班级的报告，让教师修改教学内容以改善学生的学习。这项为期五年的计划旨在推动电脑适配测验和评估发展在数学教学中的应用。该项目采用迭代和利益攸关方知情的基于科学的设计方法，并在项目开发和验证过程中使用Rasch模型进行心理测量分析。该项目的目标是：(A)为以前确定的6-8年级问题解决措施制定基准；(B)为每项措施开发、校准和验证参照标准的计算机适应性测试；(C)构建学生和班级的分数报告，以便纳入计算机适应性测试系统；以及(D)调查教师在STEM学习环境中实施、解释和使用评估和结果的能力。该项目解决了以下一组研究问题：(RQ1)什么基准绩效标准定义了每个年级在问题解决措施上的不同熟练程度？(RQ2)为计算机适应性测试题库开发的新问题解决措施项目的心理测量学特性是什么？(RQ3)在新的问题解决测量项目上，学生群体中是否存在显著的项目漂移？(RQ4)在计算机自适应测试系统中，问题解决措施项目校准的稳定性有多大？(RQ5)教师和学生对新的问题解决措施项目、计算机自适应测试平台和报告系统有什么改进建议？(RQ6)教师在多大程度上与生成的评估信息进行互动、感知和理解，以便在实践中使用？和(RQ7)在线学习模块是否培养教师在STEM学习环境中解决问题、实施计算机自适应测试、解释和使用学生评估结果的能力？实验设计将被用来调查教师在计算机适应性测试系统中实施、解释和使用问题解决措施的能力。该项目为学区和研究人员提供了一种手段，可以高效有效地评估学生的数学问题解决成绩或随时间增长的情况；满足未来的在线学习需求；通过更准确地了解学生的优势信息，以更少的课堂时间专注于评估，从而改善课堂教学。探索研究预科-12项目(DRK-12)旨在通过研究和开发创新资源、模型和工具，显著提高预科-12年级学生和教师的科学、技术、工程和数学(STEM)的学习和教学。DRK-12计划中的项目建立在STEM教育的基础研究和先前的研究和开发工作的基础上，为拟议的项目提供了理论和经验上的证明。该奖项反映了NSF的法定使命，并通过使用基金会的智力优势和更广泛的影响审查标准进行评估，被认为值得支持。

项目成果

Validating the use of s tudent‐level instruments to examine preservice teachers' mathematical problem solving

• DOI: 10.1111/ssm.12558
• 发表时间: 2022-12
• 期刊: School Science and Mathematics
• 影响因子: 1.1
• 作者: Timothy D. Folger;Maria Stewart;J. Bostic;Toni A. May

FOURTH-GRADE STUDENTS’ SENSEMAKING OF WORD PROBLEMS

四年级学生——理解单词问题

• 发表时间: 2021
• 期刊: Proceedings for the 48th Annual Meeting of the Research Council on Mathematics Learning
• 作者: Matney, G.

Examining the influence of COVID-19 on elementary mathematics standardized test scores in a rural ohio school district.

检查 COVID-19 对俄亥俄州农村学区小学数学标准化考试成绩的影响。

• 发表时间: 2022
• 期刊: Midwestern educational researcher
• 作者: Bright, D.;Fan, Y.;Fornara, C.;Koskey, K.;Bostic, J.;May, T.;Swineford, D.
• 通讯作者: Swineford, D.

Changing populations: Using the PSMs with teachers.

不断变化的人口：与教师一起使用 PSM。

• 发表时间: 2022
• 期刊: Psychology of Mathematics Education North America
• 作者: Bostic, J.;Folger, T.;Matney, G.;May, T.;Koskey, K.;Stone, G.
• 通讯作者: Stone, G.

Examining how using dichotomous and partial credit scoring models influence sixth‐grade mathematical problem‐solving assessment outcomes

检查使用二分法和部分信用评分模型如何影响六年级数学问题 - 解决评估结果

• DOI: 10.1111/ssm.12570
• 发表时间: 2023
• 期刊: School Science and Mathematics
• 影响因子: 1.1
• 作者: May, Toni A.;Koskey, Kristin L. K.;Bostic, Jonathan D.;Stone, Gregory E.;Kruse, Lance M.;Matney, Gabriel
• 通讯作者: Matney, Gabriel