Developing Authentic and Fair Computer Science Assessments

点击翻译按钮获取中文摘要

This project aims to promote equitable design of Computer Science (CS) assessment in secondary and post-secondary education in the United States and globally, increasing the diversity of students engaging in CS learning through reduced test bias. In this study, we aim to address difficulties in assessing computer programming by investigating critical characteristics of programming tasks using both response process and product data. Findings will have direct practical implications for developing authentic, fair and valid assessment of learners with different demographic backgrounds. Through advancing our understanding of the cognitive processes underlying programming thereby informing ways to better teach, learn, and assess programming skills, we expect the project to impact the broader CS education community through shareable data sets to the general public, assessment innovations in large CS classrooms, actionable insights on test bias for CS instructors, and the engagement of undergraduates of diverse gender, race, and ability. The research team from the University of Washington also intends to integrate scientific discoveries from this study into the university’s publicly available course materials. The planned dissemination will maximize outreach to various outlets such as the NSF-supported Exploring Computing Education pathways that brings together state leaders shaping U.S. K-12 CS Education curricula, practices, and standards.This project consists of foundational research on assessing, learning and teaching computer programming skills. The project will capitalize on the ability of recording the coding process via keystroke logs to extract and summarize vast amounts of fine-grained information captured by observing program edits. We aim to study the relations between process and task characteristics, identifying patterns that are indicative of proficiencies and suggest fluency or dysfluency. Such identification will, in turn, allow for designing instructional, learning, or assessment materials that are targeted at specific needs of learners. We plan to triangulate different types of student data to address research questions around detecting meaningful behavioral patterns from timing and process data when students are engaged with computer programming, relations between tasks characteristics and programming process, student knowledge, attitudes, experience and proficiency, as well as the extent to which task design contribute to the performance patterns detected for students that vary along gender, ethnicity, and native language. The project will use controlled experiments and cognitive interviews to collect quantitative and qualitative data. Multiple instruments will be used for data collection, such as the ETS Major Field Test-Computer Science. In terms of data analysis, the project will leverage various analytical and modeling techniques from the fields of psychometrics, statistics, machine learning, and educational data mining. Findings from this project will offer empirically-tested guidelines on which task characteristics to account for when designing fair and valid assessments across different demographic groups. This project is funded by the EHR Core Research (ECR) program, which supports work that advances fundamental research on STEM learning and learning environments, broadening participation in STEM, and STEM workforce development.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.