A Validated Scoring Rubric for Explain-in-Plain-English Questions
A Validated Scoring Rubric for Explain-in-Plain-English Questions
复制标题
用于用简单英语解释问题的经过验证的评分标准
DOI：
10.1145/3328778.3366879
复制
发表时间：
2020
期刊：
影响因子：
--
通讯作者：
C. Zilles
中科院分区：
文献类型：
--
作者：
Binglin Chen;Sushmita Azad;Rajarshi Haldar;Matthew West;C. Zilles
先前的研究已经确定，阅读代码并理解其高级目的的能力是一项重要的发展技能，这比在头脑中执行给定输入的代码（“代码跟踪”）更难（对于给定的代码段），但比编写代码更容易。先前的工作涉及代码阅读（“用简单的英语解释”）问题，使用了受SOLO分类法启发的评分标准，但我们发现它很难使用，因为它不能充分处理答案质量的三个维度：正确性、抽象水平和模糊性。在本文中，我们描述了一个7分的评分标准，我们为学生对“用简单的英语解释”问题的回答打分，我们通过四种方式验证了这个评分标准。首先，我们发现该量表可以可靠地应用与中位数Krippendorff's alpha（评级间信度）为0.775。其次，我们报告了一个实验来评估我们的量表的有效性。第三，我们发现由12个代码阅读问题组成的调查具有很高的内部一致性（Cronbach’s alpha = 0.954）。最后，我们发现，在大量招生（N = 452）数据结构课程中，我们的代码阅读问题分数与代码编写性能的相关性（Pearson’s R = 0.555）与之前的研究结果相似。
Previous research has identified the ability to read code and understand its high-level purpose as an important developmental skill that is harder to do (for a given piece of code) than executing code in one's head for a given input ("code tracing"), but easier to do than writing the code. Prior work involving code reading ("Explain in plain English") problems, have used a scoring rubric inspired by the SOLO taxonomy, but we found it difficult to employ because it didn't adequately handle the three dimensions of answer quality: correctness, level of abstraction, and ambiguity. In this paper, we describe a 7-point rubric that we developed for scoring student responses to "Explain in plain English'' questions, and we validate this rubric through four means. First, we find that the scale can be reliably applied with with a median Krippendorff's alpha (inter-rater reliability) of 0.775. Second, we report on an experiment to assess the validity of our scale. Third, we find that a survey consisting of 12 code reading questions had a high internal consistency (Cronbach's alpha = 0.954). Last, we find that our scores for code reading questions in a large enrollment (N = 452) data structures course are correlated (Pearson's R = 0.555) to code writing performance to a similar degree as found in previous work.
理解第一：评估用于 CS1 程序跟踪的新颖教学法和辅导系统
DOI：
10.1145/3105726.3106178
发表时间：
2017
期刊：
ACM International Computing Education Research Conference
影响因子：
--
作者：
Nelson, Greg L.;Xie, Benjamin;Ko, Andrew J.
通讯作者：
Ko, Andrew J.
影响因子：
2.7
作者：
Xie, Benjamin;Loksa, Dastyni;Nelson, Greg L.;Davidson, Matthew J.;Dong, Dongsheng;Kwik, Harrison;Tan, Alex Hui;Hwa, Leanne;Li, Min;Ko, Andrew J.
通讯作者：
Ko, Andrew J.