The Potential of Empirically Derived Rating Scales for Inexperienced Raters: A Comparative Study on Rating Scales
The Potential of Empirically Derived Rating Scales for Inexperienced Raters: A Comparative Study on Rating Scales
复制标题
针对缺乏经验的评估者的经验派生评级量表的潜力：评级量表的比较研究
DOI：
10.20622/jltajournal.21.0_141
复制
发表时间：
2018
期刊：
影响因子：
--
通讯作者：
Keisuke Kubota
中科院分区：
文献类型：
--
作者：
Keisuke Kubota
同伴互评作为一种替代性的评价工具在英语课堂上非常流行。它也被认为是积极和协作学习的有用方法。然而，存在评分者偏见的问题（Farrokhi，Esfandiari，& Schaefer，2012; Matsuno，2009），这可能会在同行评估的情况下造成负面后果。避免这种不公平现象的一种方法是设计一个更好的衡量尺度，对同行评估的参与者等没有经验的评估者和有经验的评级者同样有效。因此，本研究的目的是探讨潜在的采用评估工具，可以帮助减轻评分员偏见同行评估。为了做到这一点，作者采用了经验得出的，二元选择，边界定义（EBB）的评级量表。本研究采用多方面Rasch测量法（MFRM），分析了由5名日本评分员（3名有经验和2名无经验）评分的45组作文数据。评价工具被用来调查评分者的严重性或宽大和评分者的偏见模式相比，两种类型的评级量表。结果显示，尽管使用了评级量表，但与先前研究中发现的评级者偏倚模式相同（Farrokhi等人，2012; Matsuno，2009）。此外，本研究还证实了经验不足的评分员的语言能力会影响他们的评分倾向。换句话说，如果一个人的语言熟练程度相对较高，那么一个没有经验的评分者可能会表现得像一个有经验的评估者。基于本研究的发现，本文讨论了在英语课堂写作评估中使用经验性评分量表的意义。
Peer assessment is currently popular as an alternative evaluation tool in the context of EFL classrooms. It is also recognized as a useful method for active and collaborative learning. There is, however, the issue of rater bias (Farrokhi, Esfandiari, & Schaefer, 2012; Matsuno, 2009), which can cause negative consequences in instances of peer assessment. One way to avoid such unfairness is to design a better measurement scale that would work equally well for inexperienced evaluators such as the participants of peer assessments and for experienced raters. This study thus aims to explore the potential of employing an assessment instrument that can help to mitigate rater bias in peer assessment. To do this, the author adopted an empirically derived, binary-choice, boundary-defined (EBB) rating scale. The investigation used the Many-Facet Rasch Measurement (MFRM) to analyze 45 sets of essay writing data scored by 5 Japanese raters (3 experienced and 2 inexperienced). The evaluation tool was employed to investigate rater severity or leniency and rater bias patterns in comparison with two types of rating scales. The results revealed that despite the utilization of the rating scales, the same patterns of rater bias occurred as were found in previous studies (Farrokhi et al., 2012; Matsuno, 2009). Moreover, the present study confirmed that the language ability of inexperienced raters could influence their rating tendency. In other words, an inexperienced rater may behave like an experienced evaluator if the person's language proficiency levels are relatively high. On the basis of the findings of the present study, this paper discusses the implications of the use of empirically developed rating scales in EFL classroom writing assessments.
开发供课堂使用的故事复述口语测试
DOI：
--
发表时间：
2008
期刊：
影响因子：
--
作者：
Hirai A.;Koizumi R.
通讯作者：
Koizumi R.