Developing Authentic and Fair Computer Science Assessments

该项目旨在促进美国和全球中学和大专教育中计算机科学(CS)评估的公平设计，通过减少考试偏见来增加学习计算机科学(CS)的学生的多样性。在这项研究中，我们旨在通过使用响应过程和产品数据调查编程任务的关键特征来解决评估计算机编程的困难。研究结果将对不同人口统计背景的学习者进行真实、公平和有效的评估具有直接的实践意义。通过促进我们对编程背后的认知过程的理解，从而提供更好地教授、学习和评估编程技能的方法，我们预计该项目将通过向公众共享数据集、在大型CS课堂上进行评估创新、针对CS教师的测试偏见的可行见解以及吸引不同性别、种族和能力的本科生参与，来影响更广泛的CS教育界。华盛顿大学的研究团队还打算将这项研究的科学发现整合到该大学公开可用的课程材料中。计划中的传播将最大限度地扩大到各种渠道，如NSF支持的探索计算教育路径，该计划将制定美国K-12 CS教育课程、实践和标准的州领导人聚集在一起。该项目包括评估、学习和教授计算机编程技能的基础研究。该项目将利用通过击键日志记录编码过程的能力来提取和总结通过观察程序编辑而捕获的大量细粒度信息。我们的目标是研究过程和任务特征之间的关系，找出表明熟练程度和流利或不流利的模式。这样的识别反过来将允许设计针对学习者特定需求的教学、学习或评估材料。我们计划对不同类型的学生数据进行三角划分，以解决以下研究问题：从学生从事计算机编程的时间和过程数据中检测有意义的行为模式，任务特征与编程过程之间的关系，学生的知识、态度、经验和熟练程度，以及任务设计对检测到的学生表现模式的影响程度，这些模式随性别、种族和母语的不同而不同。该项目将使用对照实验和认知访谈来收集定量和定性数据。将使用多种仪器进行数据收集，例如ETS专业实地测试-计算机科学。在数据分析方面，该项目将利用心理测量学、统计学、机器学习和教育数据挖掘领域的各种分析和建模技术。该项目的调查结果将提供经验性检验的指导方针，说明在设计不同人口群体的公平和有效评估时应考虑哪些任务特点。该项目由EHR核心研究(ECR)计划资助，该计划支持推进STEM学习和学习环境的基础研究、扩大对STEM的参与以及STEM劳动力发展的工作。该奖项反映了NSF的法定使命，并通过使用基金会的智力优势和更广泛的影响审查标准进行评估，被认为值得支持。

This project aims to promote equitable design of Computer Science (CS) assessment in secondary and post-secondary education in the United States and globally, increasing the diversity of students engaging in CS learning through reduced test bias. In this study, we aim to address difficulties in assessing computer programming by investigating critical characteristics of programming tasks using both response process and product data. Findings will have direct practical implications for developing authentic, fair and valid assessment of learners with different demographic backgrounds. Through advancing our understanding of the cognitive processes underlying programming thereby informing ways to better teach, learn, and assess programming skills, we expect the project to impact the broader CS education community through shareable data sets to the general public, assessment innovations in large CS classrooms, actionable insights on test bias for CS instructors, and the engagement of undergraduates of diverse gender, race, and ability. The research team from the University of Washington also intends to integrate scientific discoveries from this study into the university’s publicly available course materials. The planned dissemination will maximize outreach to various outlets such as the NSF-supported Exploring Computing Education pathways that brings together state leaders shaping U.S. K-12 CS Education curricula, practices, and standards.This project consists of foundational research on assessing, learning and teaching computer programming skills. The project will capitalize on the ability of recording the coding process via keystroke logs to extract and summarize vast amounts of fine-grained information captured by observing program edits. We aim to study the relations between process and task characteristics, identifying patterns that are indicative of proficiencies and suggest fluency or dysfluency. Such identification will, in turn, allow for designing instructional, learning, or assessment materials that are targeted at specific needs of learners. We plan to triangulate different types of student data to address research questions around detecting meaningful behavioral patterns from timing and process data when students are engaged with computer programming, relations between tasks characteristics and programming process, student knowledge, attitudes, experience and proficiency, as well as the extent to which task design contribute to the performance patterns detected for students that vary along gender, ethnicity, and native language. The project will use controlled experiments and cognitive interviews to collect quantitative and qualitative data. Multiple instruments will be used for data collection, such as the ETS Major Field Test-Computer Science. In terms of data analysis, the project will leverage various analytical and modeling techniques from the fields of psychometrics, statistics, machine learning, and educational data mining. Findings from this project will offer empirically-tested guidelines on which task characteristics to account for when designing fair and valid assessments across different demographic groups. This project is funded by the EHR Core Research (ECR) program, which supports work that advances fundamental research on STEM learning and learning environments, broadening participation in STEM, and STEM workforce development.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.