Collaborative Research: Expanding a National Network for Automated Analysis of Constructed Response Assessments to Reveal Student Thinking in STEM

合作研究：扩大构建反应评估自动分析的国家网络，以揭示学生在 STEM 中的思维

• 批准号: 1323022
• 负责人: Jennifer Knight
• 金额: $ 13.77万
• 依托单位: University of Colorado at Boulder
• 依托单位国家: 美国
• 项目类别: Continuing Grant
• 财政年份: 2013
• 资助国家: 美国
• 起止时间: 2013-09-15 至 2019-08-31
• 项目状态: 已结题
• 来源: https://www.nsf.gov/awardsearch/showAward?AWD_ID=1323022&HistoricalAwards=false
• 关键词: Collaborative; Research; Expanding; National; Network

This project is being conducted by a large team across 6 institutions that is building on already developed open-ended constructed response versions of well-established concept inventories that can be accurately assessed with already created computer automated analysis resources. The computer-automated analyses are able to predict human ratings of students' work on these topics and have demonstrated higher inter-rater reliability than a group of trained expert human graders. Constructed response assessments reveal more about student thinking and the persistence of misconceptions than do multiple-choice questions, but require more analysis on the part of the educator. In past work, items designed to identify important disciplinary constructs were created based on prior research. The items were then administered via online course management systems where students entered responses. Lexical and statistical analysis software was used to predict expert ratings of student responses. To date, the work has focused primarily in the fields of biology and chemistry in biological contexts. The current project is leveraging the previous research on Automated Assessment of Constructed Response (AACR), and extending the work to other institutions and other STEM disciplines. The specific goals of this project are to: 1. Create a community web portal for the Automated Assessment of Constructed Response (AACR) assessments to expand and deepen collaborations among STEM education researchers, thus providing the infrastructure for expanding the community of researchers and supporting the adoption and implementation of the innovative instructional materials by instructors at other institutions. 2. Propagate the innovations by providing instructors with professional development and long-term, ongoing support to use the assessments. This includes information about common student conceptions revealed by the questions, instructional materials for addressing conceptual barriers, and the opportunity to join a community of practitioners who are using the AACR questions and exchanging materials. 3. Expand the basic research to create and validate AACR questions in introductory chemistry, chemical engineering, and statistics. 4. Engage in ongoing project evaluation for continuous quality improvement and to document the challenges and successes the project encounters. 5. Lay the foundation for sustainability by providing interfaces for e-text publishers, Learning Management System vendors, and Massively Open Online Courses as potential revenue streams to operate and maintain the online infrastructure.Intellectual Merit:Improving STEM education requires valid and reliable instruments that provide insight into student thinking. The automated analysis of constructed response assessments have the potential to assess "big ideas" in STEM in a richer, more multi-faceted manner than multiple-choice instruments. This project is extending the number of these items and provide an online community where instructors may obtain, score, and contribute to the library of items and resources necessary for their analyses. Broader Impacts:The web portal is extending the use of the products created in this project to instructors nationwide. In addition it is providing the foundation for a national collaboration of science and engineering educators interested in developing deeper conceptual assessment tools and supports and mentors postdoctoral research fellows, and graduate research assistants in STEM education research.

该项目由 6 个机构的大型团队进行，该项目建立在已开发的开放式构建响应版本的成熟概念清单的基础上，可以使用已创建的计算机自动分析资源进行准确评估。 计算机自动分析能够预测人类对学生在这些主题上的作业的评分，并且比一组训练有素的专家评分者表现出更高的评分者间可靠性。与多项选择题相比，构建反应评估更多地揭示了学生的思维和误解的持续存在，但需要教育者进行更多的分析。在过去的工作中，旨在识别重要学科结构的项目是根据先前的研究创建的。然后，这些项目通过在线课程管理系统进行管理，学生可以在其中输入答案。词汇和统计分析软件用于预测专家对学生反应的评分。迄今为止，这项工作主要集中在生物背景下的生物学和化学领域。当前的项目正在利用之前关于构建响应自动评估 (AACR) 的研究，并将工作扩展到其他机构和其他 STEM 学科。该项目的具体目标是： 1. 创建一个用于构建反应自动评估（AACR）评估的社区门户网站，以扩大和深化 STEM 教育研究人员之间的合作，从而为扩大研究人员社区提供基础设施，并支持其他机构的教师采用和实施创新教学材料。 2. 通过为教师提供专业发展和使用评估的长期持续支持来传播创新。这包括问题揭示的常见学生概念的信息、解决概念障碍的教学材料，以及加入使用 AACR 问题和交换材料的从业者社区的机会。 3. 扩大基础研究，创建和验证化学入门、化学工程和统计学中的 AACR 问题。 4. 参与正在进行的项目评估，以持续改进质量，并记录项目遇到的挑战和成功。 5. 通过为电子文本出版商、学习管理系统供应商和大规模开放在线课程提供接口作为运营和维护在线基础设施的潜在收入来源，为可持续发展奠定基础。智力价值：改善 STEM 教育需要有效且可靠的工具来洞察学生的思维。与多项选择工具相比，构建响应评估的自动分析有可能以更丰富、更多方面的方式评估 STEM 中的“大创意”。该项目正在扩展这些项目的数量，并提供一个在线社区，教师可以在其中获取、评分并向分析所需的项目和资源库做出贡献。更广泛的影响：该门户网站正在将该项目中创建的产品的使用扩展到全国范围内的教师。此外，它还为有兴趣开发更深入的概念评估工具的科学和工程教育工作者的全国合作奠定了基础，并支持和指导 STEM 教育研究的博士后研究员和研究生研究助理。