Development of Assessment Tools for Evaluating Organic Chemistry Learners' Understanding of Reaction Mechanisms

本项目旨在通过开发和实施基于计算机的评分模型来评估学生在大学有机化学课程中对反应机制的学习，从而为国家利益服务。培养对反应机制的良好理解对于学生在有机化学课程中取得成功至关重要，这往往是学生寻求完成STEM学位和推进其专业目标的障碍。为了应对这一挑战，项目团队将设计预测评分模型，以帮助教师在学生学习期间以及在课堂上向教师提供反馈。调查人员将收集学生对他们认为正在发生的有机化学反应机制的书面解释；开发基于计算机的评分模型，使用机器学习文本分析来评估书面解释；并为有机化学教师提供专业发展机会（包括面对面和异步在线），让他们反思如何评估学生对反应机制的理解，以及如何将新技术纳入有机化学课程。来自南佛罗里达大学（USF）的项目团队将：(1)开发机器学习文本分析评分模型，(2)使用评分模型探索学生在两个学期的有机化学课程序列中对反应机制的理解是如何发展的，以及(3)为有机化学教师提供专业发展机会。这项工作将探索如何使用机器学习技术来开发化学主题的预测评分模型，例如亲核试剂或亲电试剂在有机化学反应机制中的作用。这项工作将扩展当前仅评估单个评估提示的书面解释的预测评分模型，以开发一组可用于评估各种提示的书面解释的模型。此外，这项工作将捕捉教授有机化学的教师如何参与并回应(1)学习者对反应机制中发生的事情的书面解释和(2)强调有机会提高学生对反应机制的学习的教育研究文献。该项目活动有可能对USF研究中心7000多名学生的学习和STEM保留产生积极影响，这是一个新兴的西班牙裔服务机构，拥有多样化的人口，包括调查对英语不是第一语言的学习者的不同影响。通过这项工作，项目团队还将努力鼓励其他研究人员探索更广义的预测模型，以解决STEM课程中更广泛的概念和技能（例如论证和解释）。NSF IUSE: EDU项目支持研究和开发项目，以提高所有学生STEM教育的有效性。通过参与学生学习轨道，该计划支持有前途的实践和工具的创建，探索和实施。该奖项反映了美国国家科学基金会的法定使命，并通过使用基金会的知识价值和更广泛的影响审查标准进行评估，被认为值得支持。

This project aims to serve the national interest by developing and implementing computer-based scoring models to evaluate students’ learning about reaction mechanisms in college-level organic chemistry courses. Developing a good understanding of reaction mechanisms is critical for students’ success in organic chemistry courses, which are often a barrier for students seeking to complete STEM degrees and advance their professional goals. To address this challenge, the project team will design predictive scoring models to help instructors provide feedback to students during their studies, as well as to instructors during class sessions. The investigators will collect students’ written explanations for what they think is happening for organic chemistry reaction mechanisms; develop computer-based scoring models using machine learning text analyses to evaluate the written explanations; and offer professional development opportunities (both in-person and asynchronously online) for organic chemistry instructors to reflect upon how they assess students’ understanding of reaction mechanisms and how they can incorporate the new technologies into their organic chemistry courses.The project team from the University of South Florida (USF) will: (1) develop machine learning text analysis scoring models, (2) use the scoring models to explore how student understanding of reaction mechanisms develops across the two-semester organic chemistry course sequence, and (3) facilitate professional development opportunities for organic chemistry instructors. This work will explore how machine learning technologies can be used to develop predictive scoring models for chemistry topics such as the role of nucleophiles or electrophiles in organic chemistry reaction mechanisms. The work will extend current predictive scoring models that only evaluate written explanations for a single assessment prompt to the development of a set of models that can be used to evaluate written explanations for a wide array of prompts. Additionally, this work will capture how faculty members teaching organic chemistry engage with and respond to (1) learners’ written explanations of what is happening in a reaction mechanism and (2) the education research literature that highlights opportunities to improve students’ learning of reaction mechanisms. The project activities have the potential to positively impact the learning and STEM retention of 7,000+ students at the USF study site, an emerging Hispanic-serving institution with a diverse population, including investigating differential impacts on learners for whom English is not their first language. Through this work, the project team will also strive to encourage other researchers to explore more generalized predictive models that address broader concepts and skills (e.g., argumentation and explanation) in STEM courses. The NSF IUSE: EDU program supports research and development projects to improve the effectiveness of STEM education for all students. Through the Engaged Student Learning track, the program supports the creation, exploration, and implementation of promising practices and tools.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.