Developing Authentic and Fair Computer Science Assessments

制定真实且公平的计算机科学评估

• 批准号: 2100296
• 负责人: Amy Ko
• 金额: $ 87.74万
• 依托单位: University of Washington
• 依托单位国家: 美国
• 项目类别: Continuing Grant
• 财政年份: 2021
• 资助国家: 美国
• 起止时间: 2021-10-01 至 2024-09-30
• 项目状态: 已结题
• 来源: https://www.nsf.gov/awardsearch/showAward?AWD_ID=2100296&HistoricalAwards=false
• 关键词: Developing; Authentic; Fair; Computer; Science

This project aims to promote equitable design of Computer Science (CS) assessment in secondary and post-secondary education in the United States and globally, increasing the diversity of students engaging in CS learning through reduced test bias. In this study, we aim to address difficulties in assessing computer programming by investigating critical characteristics of programming tasks using both response process and product data. Findings will have direct practical implications for developing authentic, fair and valid assessment of learners with different demographic backgrounds. Through advancing our understanding of the cognitive processes underlying programming thereby informing ways to better teach, learn, and assess programming skills, we expect the project to impact the broader CS education community through shareable data sets to the general public, assessment innovations in large CS classrooms, actionable insights on test bias for CS instructors, and the engagement of undergraduates of diverse gender, race, and ability. The research team from the University of Washington also intends to integrate scientific discoveries from this study into the university’s publicly available course materials. The planned dissemination will maximize outreach to various outlets such as the NSF-supported Exploring Computing Education pathways that brings together state leaders shaping U.S. K-12 CS Education curricula, practices, and standards.This project consists of foundational research on assessing, learning and teaching computer programming skills. The project will capitalize on the ability of recording the coding process via keystroke logs to extract and summarize vast amounts of fine-grained information captured by observing program edits. We aim to study the relations between process and task characteristics, identifying patterns that are indicative of proficiencies and suggest fluency or dysfluency. Such identification will, in turn, allow for designing instructional, learning, or assessment materials that are targeted at specific needs of learners. We plan to triangulate different types of student data to address research questions around detecting meaningful behavioral patterns from timing and process data when students are engaged with computer programming, relations between tasks characteristics and programming process, student knowledge, attitudes, experience and proficiency, as well as the extent to which task design contribute to the performance patterns detected for students that vary along gender, ethnicity, and native language. The project will use controlled experiments and cognitive interviews to collect quantitative and qualitative data. Multiple instruments will be used for data collection, such as the ETS Major Field Test-Computer Science. In terms of data analysis, the project will leverage various analytical and modeling techniques from the fields of psychometrics, statistics, machine learning, and educational data mining. Findings from this project will offer empirically-tested guidelines on which task characteristics to account for when designing fair and valid assessments across different demographic groups. This project is funded by the EHR Core Research (ECR) program, which supports work that advances fundamental research on STEM learning and learning environments, broadening participation in STEM, and STEM workforce development.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.

该项目旨在促进美国和全球中学和中学后教育中计算机科学（CS）评估的公平设计，通过减少测试偏差来增加参与CS学习的学生的多样性。在这项研究中，我们的目标是解决困难，评估计算机编程，通过调查的关键特性的编程任务，使用响应过程和产品数据。研究结果将有直接的实际意义，发展真实，公平和有效的评估不同的人口背景的学习者。通过推进我们对编程认知过程的理解，从而为更好地教授、学习和评估编程技能提供信息，我们希望该项目通过向公众提供可共享的数据集、大型CS教室的评估创新、对CS教师测试偏见的可操作见解以及不同性别、种族和能力的本科生的参与，来影响更广泛的CS教育社区。来自华盛顿大学的研究小组还打算将这项研究的科学发现整合到该大学的公开课程材料中。计划的传播将最大限度地扩大到各种渠道，如NSF支持的探索计算教育途径，汇集了国家领导人塑造美国K-12 CS教育课程，实践和标准。该项目包括评估，学习和教学计算机编程技能的基础研究。该项目将利用通过日志记录编码过程的能力，以提取和总结通过观察程序编辑捕获的大量细粒度信息。我们的目标是研究过程和任务特征之间的关系，识别模式，是熟练的指示，并建议流畅或不流畅。这种确定反过来又有助于设计针对学习者具体需要的教学、学习或评估材料。我们计划对不同类型的学生数据进行三角分析，以解决以下研究问题：从学生从事计算机编程时的时间和过程数据中检测有意义的行为模式，任务特征与编程过程之间的关系，学生的知识，态度，经验和熟练程度，以及任务设计对检测到的学生表现模式的贡献程度，这些表现模式沿着性别而变化，种族和母语。该项目将使用控制实验和认知访谈来收集定量和定性数据。多个工具将用于数据收集，如ETS专业领域测试-计算机科学。在数据分析方面，该项目将利用心理测量学、统计学、机器学习和教育数据挖掘领域的各种分析和建模技术。该项目的研究结果将提供经过实践检验的指导方针，说明在设计不同人口群体的公平有效评估时应考虑哪些任务特征。该项目由EHR核心研究（ECR）计划资助，该计划支持推进STEM学习和学习环境的基础研究，扩大STEM参与，以及STEM劳动力发展的工作。该奖项反映了NSF的法定使命，并通过使用基金会的智力价值和更广泛的影响审查标准进行评估，被认为值得支持。

项目成果

A Decade of Demographics in Computing Education Research: A Critical Review of Trends in Collection, Reporting, and Use

• DOI: 10.1145/3501385.3543967
• 发表时间: 2022-08
• 期刊: Proceedings of the 2022 ACM Conference on International Computing Education Research - Volume 1
• 作者: A. Oleson;Benjamin Xie;Jean Salac;Jayne Everson;F. M. Kivuva;Amy J. Ko