Collaborative Research: Building Assessment Capacity in Chemistry Education - The Chemistry Instrument Review and Assessment Library Project

本项目旨在促进以证据为基础的评估教学工具的采用，以服务于高质量本科化学教育的国家利益。了解教育活动对学生学习的影响需要有效的评估工具的数据，这些评估工具与感兴趣的结果相一致。本项目旨在帮助化学教师和化学教育研究者选择课堂使用的有效仪器。具体而言，该项目将开发和主持一个化学教育界使用的评估工具目录。该目录将以用户友好的格式提供，并托管在化学教育交流网站上，该网站是美国化学会于2012年建立的社区资源。为了进一步支持对高质量仪器的选择，该目录还将：1)提供有关使用仪器收集的数据的有效性和可靠性的信息；2)制定与效度和信度相关的概念和方法词汇表；3)包括针对每个评估的现有证据的专家评论。通过这项工作，化学仪器审查和评估库（CHIRAL）平台将成为化学教育界的中心资源。为了实现其目标，项目团队将系统地审查现有文献，以确定化学教育中使用的评估工具。一个专家小组将编写一份关于每种工具的摘要报告，以帮助用户根据特定情况选择最合适的评估工具。除了基本元数据（如仪器的主题和格式）外，CHIRAL平台中的每个仪器都将包括有关该仪器有效性和可靠性的证据概述。这些信息将包括来自开发人员和使用该仪器的其他研究的结果。此外，摘要和概述中包含的有效性和可靠性的关键原则将超链接到动态术语表。本术语表将总结和参考现有的高质量资源，这些资源以标准化的格式解释了从基础到高级主题的常见心理测量术语和方法。为了支持CHIRAL平台对化学教育界的价值，用户将有多种方式提供对其功能的反馈。预计CHIRAL平台收集和呈现的信息将改善用户在评估和实施化学教育评估方面的决策过程。为了衡量这种影响的程度，将进行一项定性评估研究，以衡量教师对假设工具的观点，这些假设工具有不同程度的证据来支持其数据的有效性和可靠性。在化学教育界，目录、同行评审过程和术语表的结合有可能增加学习评估过程的严谨性，提高仪器开发中的学术工作的价值，并减少教育工作者和研究人员致力于寻找或开发适合他们需要的评估的时间。因此，该项目有望提高课堂教学水平和学生学习化学的教育研究。本项目由美国国家科学基金会改善本科STEM教育计划：教育与人力资源（IUSE: EHR）支持，该计划支持研究和开发项目，以提高所有学生STEM教育的有效性。通过参与学生学习轨道，该计划支持有前途的实践和工具的创建，探索和实施。该奖项反映了美国国家科学基金会的法定使命，并通过使用基金会的知识价值和更广泛的影响审查标准进行评估，被认为值得支持。

This project aims to serve the national interest in high-quality undergraduate chemistry education by promoting the adoption of evidence-based tools for assessing instruction. Understanding the impact of an educational initiative on student learning requires data from valid assessment instruments that are aligned to the outcome of interest. This project aims to help chemistry instructors and chemistry education researchers select valid instruments for classroom use. Specifically, the project will develop and host a catalog of assessment instruments used by the chemistry education community. The catalog will be available in a user-friendly format and hosted on the Chemical Education Xchange, an American Chemical Society community resource established in 2012. To further support the selection of high-quality instruments, this catalog will also: 1) provide information about the validity and reliability of data collected using the instrument; 2) develop a glossary of the concepts and methods related to validity and reliability; and 3) include expert reviews regarding the available evidence for each assessment. Through this work, the CHemistry Instrument Review and Assessment Library (CHIRAL) platform will be developed as a central resource for the chemical education community. To meet its goals, the project team will systematically review available literature to identify assessment instruments that are used in chemistry education. A panel of experts will write a summary report about each instrument, to aid users in selecting the most appropriate assessment instrument for a given context. Along with basic metadata, such as the topic and format of an instrument, each instrument in the CHIRAL platform will include an overview of the evidence regarding the instrument's validity and reliability. This information will include results from the developers and from other studies that have used the instrument. Additionally, key principles of validity and reliability included in the summary and overview will be hyperlinked to a dynamic glossary. This glossary will summarize and reference existing quality resources that explain common psychometric terminology and methods in a standardized format that scaffolds from basic to advanced topics. To support the value of the CHIRAL platform to the chemistry education community, users will have multiple ways to provide feedback on its features. The information gathered and presented by the CHIRAL platform is predicted to improve users' decision-making processes in evaluating and implementing educational assessments in chemistry. To measure the extent of this impact, a qualitative evaluation study will be conducted to gauge faculty's perspectives of hypothetical instruments supported with varying levels of evidence for the validity and reliability of their data. Within the chemistry education community, the combination of the catalog, peer-review process, and the glossary have the potential to increase the rigor of the learning assessment process, enhance the value of scholarly work in instrument development, and reduce the time educators and researchers dedicate to locating or developing appropriate assessments for their needs. As a result, this project is expected to enhance both instruction at the classroom level and educational research on student learning in chemistry. This project is supported by the NSF Improving Undergraduate STEM Education Program: Education and Human Resources (IUSE: EHR), which supports research and development projects to improve the effectiveness of STEM education for all students. Through the Engaged Student Learning track, the program supports the creation, exploration, and implementation of promising practices and tools.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.