Developing Authentic and Fair Computer Science Assessments

本项目旨在促进美国和全球中等和高等教育中计算机科学（CS）评估的公平设计，通过减少测试偏见来增加参与CS学习的学生的多样性。在这项研究中，我们的目标是通过使用响应过程和产品数据调查编程任务的关键特征来解决评估计算机编程的困难。研究结果将对不同人口背景的学习者进行真实、公平和有效的评估具有直接的实际意义。通过提高我们对编程背后认知过程的理解，从而为更好地教授、学习和评估编程技能提供方法，我们希望该项目能够通过向公众共享数据集、评估大型CS教室的创新、CS教师对测试偏见的可操作见解以及不同性别、种族和能力的本科生的参与，影响更广泛的CS教育界。来自华盛顿大学的研究小组还打算将这项研究的科学发现整合到该大学的公开课程材料中。计划中的传播将最大限度地扩展到各种渠道，如nsf支持的探索计算教育途径，该途径汇集了国家领导人塑造美国K-12计算机科学教育课程，实践和标准。本项目包括对计算机编程技能的评估、学习和教学的基础研究。该项目将利用通过击键日志记录编码过程的能力，提取和总结通过观察程序编辑捕获的大量细粒度信息。我们的目的是研究过程和任务特征之间的关系，找出表明熟练程度和表明流利或不流利的模式。这样的识别反过来又允许设计针对学习者特定需求的教学、学习或评估材料。我们计划对不同类型的学生数据进行三角测量，以解决以下研究问题：当学生从事计算机编程时，从时间和过程数据中检测有意义的行为模式，任务特征与编程过程之间的关系，学生的知识，态度，经验和熟练程度，以及任务设计对根据性别，种族和母语变化的学生检测到的表现模式的贡献程度。该项目将使用控制实验和认知访谈来收集定量和定性数据。多种仪器将用于数据收集，如ETS主要现场测试-计算机科学。在数据分析方面，该项目将利用心理测量学、统计学、机器学习和教育数据挖掘领域的各种分析和建模技术。该项目的研究结果将提供经过经验检验的指导方针，指导在针对不同人口群体设计公平有效的评估时应考虑哪些任务特征。该项目由EHR核心研究（ECR）计划资助，该计划支持推进STEM学习和学习环境基础研究、扩大STEM参与和STEM劳动力发展的工作。该奖项反映了美国国家科学基金会的法定使命，并通过使用基金会的知识价值和更广泛的影响审查标准进行评估，被认为值得支持。

This project aims to promote equitable design of Computer Science (CS) assessment in secondary and post-secondary education in the United States and globally, increasing the diversity of students engaging in CS learning through reduced test bias. In this study, we aim to address difficulties in assessing computer programming by investigating critical characteristics of programming tasks using both response process and product data. Findings will have direct practical implications for developing authentic, fair and valid assessment of learners with different demographic backgrounds. Through advancing our understanding of the cognitive processes underlying programming thereby informing ways to better teach, learn, and assess programming skills, we expect the project to impact the broader CS education community through shareable data sets to the general public, assessment innovations in large CS classrooms, actionable insights on test bias for CS instructors, and the engagement of undergraduates of diverse gender, race, and ability. The research team from the University of Washington also intends to integrate scientific discoveries from this study into the university’s publicly available course materials. The planned dissemination will maximize outreach to various outlets such as the NSF-supported Exploring Computing Education pathways that brings together state leaders shaping U.S. K-12 CS Education curricula, practices, and standards.This project consists of foundational research on assessing, learning and teaching computer programming skills. The project will capitalize on the ability of recording the coding process via keystroke logs to extract and summarize vast amounts of fine-grained information captured by observing program edits. We aim to study the relations between process and task characteristics, identifying patterns that are indicative of proficiencies and suggest fluency or dysfluency. Such identification will, in turn, allow for designing instructional, learning, or assessment materials that are targeted at specific needs of learners. We plan to triangulate different types of student data to address research questions around detecting meaningful behavioral patterns from timing and process data when students are engaged with computer programming, relations between tasks characteristics and programming process, student knowledge, attitudes, experience and proficiency, as well as the extent to which task design contribute to the performance patterns detected for students that vary along gender, ethnicity, and native language. The project will use controlled experiments and cognitive interviews to collect quantitative and qualitative data. Multiple instruments will be used for data collection, such as the ETS Major Field Test-Computer Science. In terms of data analysis, the project will leverage various analytical and modeling techniques from the fields of psychometrics, statistics, machine learning, and educational data mining. Findings from this project will offer empirically-tested guidelines on which task characteristics to account for when designing fair and valid assessments across different demographic groups. This project is funded by the EHR Core Research (ECR) program, which supports work that advances fundamental research on STEM learning and learning environments, broadening participation in STEM, and STEM workforce development.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.