BCSER: Developing Expertise in Educational Testing to Support Characterizing the Impact of Organic Chemistry Instruction on Students' Ability to Reason with Representations

具象能力是指在特定领域（如有机化学）使用绘图、图表和符号来学习、交流和解决问题的能力。研究表明，大学生运用具体表述进行推理的能力对他们理解有机化学并取得成功至关重要。然而，目前还没有有效的评估工具来测量学生在有机化学中的表征能力。在美国国家科学基金会STEM教育研究EHR核心研究能力建设（ECR: BCSER）计划的支持下，该项目旨在开发这样一种仪器。在进行这个项目的过程中，首席研究员将发展教育测试和心理测量学方面的能力。这些新的能力将补充现有的专业知识，对学生对化学概念的理解进行定性调查，并对教师的教学信念和课堂实践进行表征。项目团队开发的高质量评估工具有望对有机化学学习的进一步研究有用，并有助于大学化学教师对其教学做出明智的决定。本研究将经典测试理论与因子分析方法相结合，开发并测试有机化学表征能力评估（ORCA）。ORCA将测量学生在有机化学表征的背景下的表征能力技能，如横线楔形图、纽曼投影和椅子构象。ORCA测量的目标技能包括：(a)解释这些表征，(b)为特定目的选择适当的表征，(c)在这些表征之间建立联系，以及(d)使用这些表征对化学现象进行预测的能力。测量质量框架，其目标是由教育和心理测试标准提供的心理测量证据，将被用作评估ORCA项目质量的概念和方法框架，以及它有助于得出关于学生表征能力技能的有效和可靠推论的能力。来自北卡罗莱纳大学格林斯博罗分校和南佛罗里达大学的有机化学学生将对该仪器进行严格的评估和测试。首席研究员将由本科化学课程内容评估领域的专家指导，并将完成课程作业，以提高心理测量学和测量学的熟练程度。项目的成功和项目目标的实现将通过与导师和咨询委员会的定期会议进行评估。项目工作的传播将通过在学术期刊上的出版物、在有关会议上的发言以及关于评估、测量、设计和评价的讲习班来促进。本研究的结果将为有机化学课程的教学实践和基于证据的课程设计提供信息，以支持使用有效的策略来培养学生的表征能力。该项目得到了EHR核心研究：STEM教育研究能力建设（ECR: BCSER）竞赛的支持，该竞赛旨在培养个人在STEM学习和学习环境中开展高质量的基础STEM教育研究的能力，扩大STEM领域的参与，以及STEM劳动力的发展。该奖项反映了美国国家科学基金会的法定使命，并通过使用基金会的知识价值和更广泛的影响审查标准进行评估，被认为值得支持。

Representational competence is the ability to use drawings, diagrams, and symbols to learn, communicate, and solve problems in a specific area, such as organic chemistry. Studies have shown that undergraduate students’ ability to reason with specific representations is important for their understanding of and success in organic chemistry. However, no validated assessment instruments currently exist that measure students’ representational competence in organic chemistry. With support from NSF’s EHR Core Research Building Capacity in STEM Education Research (ECR: BCSER) program, this project seeks to develop such an instrument. In the process of conducting this project, the principal investigator will develop competencies in educational testing and psychometrics. These new competencies will complement existing expertise in qualitative investigation of students’ understanding of chemistry concepts and characterization of instructors’ teaching beliefs and classroom practices. The high-quality assessment instrument developed by the project team is expected to be useful for further research on organic chemistry learning, as well as for helping college chemistry instructors make informed decisions about their teaching.This study will use a combination of classical test theory and factor analysis methods to develop and test the Organic-chemistry Representational Competence Assessment (ORCA). ORCA will measure student representational competence skills in the context of organic chemistry representations such as dash-wedge diagrams, Newman projections, and chair conformations. The targeted skills to be measured by ORCA include: the ability to (a) interpret these representations, (b) select an appropriate representation for a particular purpose, (c) make connections across these representations, and (d) use these representations to make predictions about chemical phenomena. The measurement quality framework, which targets select psychometric evidence informed by the Standards for Educational and Psychological Testing, will be used as the conceptual and methodological framework for evaluating the quality of ORCA’s items and its capacity to assist in drawing valid and reliable inferences about student representational competence skills. The instrument will be rigorously evaluated and tested with organic chemistry students from the University of North Carolina at Greensboro and the University of South Florida. The principal investigator will be mentored by an expert in the area of assessment of content in undergraduate chemistry courses, and will complete coursework to develop proficiency in psychometrics and measurement. The success of the project and achievement of project objectives will be assessed through periodic meetings with both the mentor and the advisory board. Dissemination of project work will be facilitated via publications in academic journals, presentations at relevant conferences, and workshops on assessment measurement design and evaluation. Results of this study will inform instructional practices and evidence-based design of organic chemistry curricula that supports the use of effective strategies to develop student representational competence. This project is supported through the EHR Core Research: Building Capacity in STEM Education Research (ECR: BCSER) competition that is designed to build individuals’ capacity to carry out high-quality, fundamental STEM education research in STEM learning and learning environments, broadening participation in STEM fields, and STEM workforce development.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.